Introducing the Model Factory
TL;DR: Traditional foundation model training approaches require substantial manual interaction, little signal for improvement, and slow iteration times. To resolve these issues, we built the Model Factory, Reinforce's internal systems framework for quickly training, scaling, and experimenting with novel foundation models. In this post, we share our methodology and reasoning for building the Model Factory.
Introduction
The world of AI is in the midst of a Cambrian explosion: new models, ideas, and techniques surface daily, and keeping up has become a full-time job. New ideas are developed based on intuition, deployed on supercomputer-scale clusters and pushed into production long before the associated theory has caught up.
Organizations typically scale along two axes: scaling with people, and scaling with engineering. Hiring top-tier engineers and researchers can push the boundaries of what is possible, but this requirement clashes with the traditional, linear approach to training models whereby substantial amounts of engineering time was typically dedicated to manually handling training runs.
A Tour of the Model Factory
In order to explain why the Model Factory enables us to handle combinatorial complexity, we first need to understand how it evolved. Let's consider the following scenario: imagine we want to train a foundation model as optimally as possible.
Iteration One: Traditional Foundation Model Training
As a first approximation, we'll start by training some pre-determined architecture on a pre-determined dataset. We load data, run the model's forward pass, compute loss, run backwards pass, and repeat.
This is a good first approach, but it has many shortcomings. Everything is pre-determined. We have no way of knowing how this model could be better until we are done.
Iteration Two: Automated Model Evaluations
Let's take another look at the pipeline. The first thing we're missing is a rigorous way to automatically evaluate the model's ability on certain downstream tasks.
