OLMo & Friends: Chronicles from Building Fully Open Language Models

The explosive progress of language models in recent years has been accompanied by a lack of transparency about how these transformative technologies are created. AI labs with the resources to train such models share few details of their development pipelines, even when the resulting models are openly distributed. This opacity makes conducting research on these models challenging and limits overall progress in the field. In this talk, I will discuss our journey in building OLMo (https://allenai.org/olmo), a fully open language model. Unlike other language models, every component of OLMo—including training data, code, intermediate checkpoints, and more—is transparently released for researchers. I will highlight recent releases such as Tulu 3, OLMo 2, and OLMoE, and share insights from our two-year journey.