Today, we celebrate the 3rd birthday of the neolab Tom, Luk and I started.
As a neolab, we never made a single hire. We were a team of 3 people, training an exotic architecture on petabytes of data across clusters of thousands of, primarily non-NVIDIA, GPUs. Thrown straight into the deep end, we stumbled but learned in the first few months more about frontier-scale infra and model training than any other education could have ever provided us.
We learned most lessons the hard way, usually by breakings things at a scale where breaking things costs a lot of money very quickly.
Those lessons, as well as the infra, compilers and tooling we built to support our own research endeavours form the basis of what we now call Model Foundry.
The pivot
In April 2025, I spent 2 weeks in having many conversations with neo-lab founders and researchers and one pattern recurred over and over again: people cared a lot more about how 3 people iterated at that pace and scale than they did about the models themselves.
A specific coffee conversation still sticks out to me today as the pivotal moment where it became clear (even if it still took a bit to make the jump) that we should be focusing on the infra. I was talking to a neolab founder who had raised hundreds of millions of dollars and had several thousand GPUs reserved, which sounds like the dream, and yet they were still blocked by kernels, infra and failures. They clearly had the money, they had the ability to recruit and they had the compute, but they didn't have the infra to turn either into results at the speed they had hoped for.
It became clear to us that we would have far more impact resolving those roadblocks than being "one of n" neolabs. Being "one of n" neolabs is a perfectly good thing to be, but making n bigger seemed like a better one.
3 years in numbers
Over the past 3 years, we've trained a lot of models and ran a lot of runs for friends & customers. To be precise, we're talking about over 10,000 training runs and experiments on clusters ranging from single nodes to several thousand GPUs.
Across those runs, we've run and optimized training for more than 100 unique architectures (not in the "we swapped out the activation function" sense, but the fundamentally different architecture sense), spanning drug discovery, material science, medical AI, robotics, genomics, image generation & editing, video generation and voice models.
Looking at the hardware side, we've worked on NVIDIA, AMD, TPU, Trainium as well as 5 other vendors, across 23 microarchitectures, several dozen of SKUs and more than 30 clouds and providers. The majority of that compute was non-NVIDIA, which as far as we know puts us in a very tiny club.
Why infra
When we strip down everything to the one number that determines how fast a research lab moves and how likely they are to succeed in making a big innovative jump, it's iteration time. The faster you can go from an idea to a result, the more ideas you get to test, and the more likely you are to find the one that works.
The only reason our own thousands of runs were possible was because we spent an unreasonable amount of time making each one cheaper, faster and able to recover when it fails.
Everything else follows from there. A shorter iteration time means a smaller team can compete, which lowers the barrier to start a neolab in the first place. Faster kernels means less money spent per result and more out of the compute you have. Cross-vendor compatibility allows you to target hardware with better availability and not get held hostage by capacity crunches. And architectures that would have been too slow or too fragile to train at scale become something you can actually try.
If we multiply that across every neolab and every would-be neolab founder who hasn't taken the leap yet, then it starts to change what the future of AI looks like and how fast we get there, because more labs training more kinds of models means AI diffuses into the rest of the world faster. Right now the future of what AI looks like and which capabilities are prioritized and researched is largely decided by a handful of large LLM labs, so every neolab that can afford to exist, iterate faster and train something the big labs wouldn't is another path for capabilities that otherwise may not exist coming to fruition.
Model Foundry
Over the past 3 years, we've built 57 internal tools covering everything from cluster planning and bare metal provisioning to kernel optimization, intra-kernel debugging and experiment management. Almost every neolab ends up building some version of many of these tools, but almost no one invests into turning them into something clean and maintained. They start as duct-tape solutions that you swear you'll get around to fixing at some point, but then things pile up on the infra team's backlog and no one ever does.
We had the same problem, but with one difference: building infra for neolabs is now our core business, so cleaning up those tools is the job. We spent the last 1.5 years building a single control plane for research, which we call Model Foundry. The goal is for it to be the single surface that a researcher has to interact with day-to-day to go from idea to frontier model.
One loop
When we think about the research process, it often looks a lot more like art than science and things are scattered all over the place, but when you actually think about it, it's really all a single loop: you train a model, you evaluate it, you deploy it and you take the learnings from what it gets wrong in production to train the next model. When we actually look at what that process looks like today though, we have dozens of tools and services we stitch together, each covering a single step and we lose a lot of context at every step that needs to be retained and rebuilt by hand.
With Model Foundry we take it a step further. When you actually break down the training and experimental process into the individual steps, what you see is: raw data, pre-processing steps that take some raw data to create datasets, mixers combining those datasets, different experiments that pull mixing results and produce checkpoints, evals run on those checkpoints. That is, in essence, just a directed acyclic graph (DAG) and representing the entire research process as a DAG gives us a lot of benefits.
For example, you can take a dataset and see all the experiments that used it. You can take an eval and see the checkpoints it used, go up into the runs that produced it and then the datasets that it used giving you the ability to investigate things better, avoid duplicate effort and have true end-to-end lineage across the entire research process.
Built from the things we broke
A lot of Model Foundry is a direct consequence of the lessons we learned the hard and expensive way.
Especially as a young lab, finding capacity is one of the most difficult things, which is why we designed everything from the ground up to ensure that the same code and config run on NVIDIA, AMD, TPU or Trainium runs the same, so that every run can go wherever capacity is available and cheapest instead of waiting on a single vendor.
And because all of it sits on top of our own training engine, kernels and fleet, you get the benefit of years of optimization work without changing a line of code.
The result is that the only thing standing between a researcher and their next model is their next idea.
