Today, compute comes in three classes. On-demand is reliable and expensive, Reserved requires you forecast and commit to your compute spend whether you end up using it or not and Spot, sometimes called Preemptible, is up to 90% cheaper, but can be reclaimed at a moments notice and whatever was running dies with it.
Today, we're introducing a fourth class of compute: resumable compute. It costs 70% less than on-demand and your workload is never killed. When we need the capacity back, we suspend your process in-flight and resume it somewhere else, at the exact instruction where it stopped, without the program having to account for it.
Spot only works for workloads built to die
For tasks such as batch jobs, stateless inference endpoints or small jobs that can retry, spot is great. For everything else though, there is substantial engineering effort involved in shaping your workloads around spot.
The truth is that making a long-running workload spot-safe means writing checkpointing logic, handling retries, rebuilding connections and accepting lost work when eviction lands between checkpoints. Most of the teams we spoke with decided it wasn't worth it and stuck to on-demand and reserved capacity across their fleets.
Suspended, never lost
Spot's contract is simple: we may kill your workload at any time. With resumable compute we change a single word: we may kill suspend your workload at any time.
When we need to pull capacity back into our fleet, your instance's registers, memory, open files, and network connections move to another host, in another region or another cloud if needed. If no capacity is free, the instance is written to disk and resumed as soon as capacity returns.
You give up guaranteed wall-clock continuity, but you never give up work. The processes inside your instance never notice they were suspended.
| On-demand | Reserved | Spot | Resumable | |
|---|---|---|---|---|
| Price vs on-demand | 100% | 40% | 30% | 35% |
| Billing | Per Hour | 1-3 year | Per Second | Per Second |
| When capacity is reclaimed | Never reclaimed | Never reclaimed | Killed | Suspended, then resumed |
| Work lost on interruption | None | None | Since last checkpoint | None |
| Code changes needed | None | None | Checkpoints, retries | None |
How it works
For the most straightforward demonstration, consider a program that counts to 20, with no checkpoints and no idea that it might move.
cint main() {
while (i < 20) { printf("%d ", i++); }
}
Running it on a resumable compute instance, the output shows counting 0,1,2,...,17,18,19. What happened is that elements 0-4 printed on a host in eu-central, then the program was moved to ap-southeast where 5-9 were printed, before being moved to us-west where 10-14 were printed and then the final 15-19 were printed back on the original eu-central host. Despite running across three hosts, in three clouds across three parts of the planet and without the program being aware, the output is identical to a run on a single machine.
When moving compute between hosts without the program noticing, one of the largest problems you run into is connections.
A socket to a specific host is not able to outlive the host if the host owns it, which is why we run an ultra high-performance persistent proxy at our edge that holds every external open, both the inbound and outbound, and re-points the internal leg to wherever the instance currently resides without loosing a single packet. This allows a download started on one host to finish on another, with the migration looking like a pause or stutter from the outside instead of a dropped connection.
A real training run
We ran a multi-node pre-training for a model and moved it between clusters 13 times along the way, about once every three minutes (even though in production, a move occurs on average every 41 hours), across 5 clusters on 3 clouds. The training script had no checkpointing and no idea it was moving, we used a normal training script and ran it as-is on resumable compute. Despite being moved around, loss converged the same as it did staying on a single cluster.
Who it's for
Resumable compute is built for workloads that are long-running, stateful, and painful to restart: training runs, data pipelines, notebooks and dev environments, and long-running evals. In short, every workload that can tolerate an occasional pause but not a restart is a great candidate for resumable compute.
Resumable compute is available today in beta on Model Foundry.
