Can Foundation Models, which rely on massive parameter counts, data sets, and compute, and have proven extremely powerful in computer vision and natural language systems, be built for High Energy Physics? To do so, several challenges must be addressed. First, foundation models are pre-trained on vast and often unlabeled data sets to learn general purpose representations. These models can then be efficiently fine-tuned with small data sets to address a wide array of down-stream tasks. However, these pre-training procedures are data-type specific and so new methods must be developed for HEP data. Second, these models represent a scale in both model size and data size that have not been explored in HEP. In this talk, we will discuss our first steps towards building HEP foundation models. We will explore how Self-Supervised pre-training methods can be adapted for HEP data, how pre-trained models can encode general knowledge of high utility when adapted for a variety of tasks, and how these methods may help mitigate uncertainties through pre-training directly on experimental data.