The first thing every team does when the GPU bill gets uncomfortable is look at spot pricing. The discount looks enormous, often 60-70% off on-demand, and the arithmetic seems obvious: move the training job to spot, save most of the money.
Then the job gets interrupted twice in an afternoon, and someone discovers that a distributed job with a lost rank does not resume, it restarts. The discount evaporates into recomputation, and the team goes back to reserved capacity convinced that spot is a trap.
Neither instinct is right. Spot is a good deal for some workloads and a bad deal for others, and the variable that separates them is not the discount, it is the duty cycle and the checkpoint cost.
Part One: The crossover
Model it properly. A reserved instance costs a fixed amount per hour whether you use it or not. A spot instance costs less per hour but has an interruption probability, and each interruption costs you the work since the last checkpoint plus the restart time.
My model uses a 40% average spot discount, 6% hourly interruption probability, and a 90-second checkpoint overhead on a 40 GB state. The result: below roughly 65% duty cycle, spot is cheaper per GPU-hour delivered. Above it, reserved wins, and the gap widens quickly because interruptions are proportional to time, not to useful work.
65% is the number to remember, and almost nobody measures their actual duty cycle before choosing. Teams assume 100% because that is what the capacity is for, then run at 40% because of queueing, data loading and idle time between experiments.
Part Two: Checkpointing is the only real control
The discount is not a lever. The interruption rate is not a lever. The checkpoint interval is, and it is the one parameter teams set by habit rather than arithmetic.
Checkpoint every thirty minutes and an interruption costs you fifteen minutes of work on average, plus restart. Checkpoint every two minutes and an interruption costs you one, but you write 40 GB every two minutes, which is 1.2 TB an hour of object storage traffic and a measurable hit on the training loop.
At the numbers above, the lost GPU-minutes per hundred hours fall from 900 with no checkpointing to 85 at a two-minute interval, while the checkpoint write overhead rises from zero to 26 minutes per hundred hours. The sum has a minimum near two minutes. That minimum is where you should be, and it is not where most jobs are.
The write overhead is the part that surprises people. A 40 GB checkpoint at 500 MB/s takes 80 seconds, which is longer than the interval you wanted. The fix is incremental or sharded checkpoints, and if your framework does not support them, that is a stronger argument against the framework than any benchmark.
Part Three: The workloads that should never be on spot
Three categories.
Long single-GPU jobs with no checkpoint support. If the job cannot resume, the expected cost of an interruption is the whole job. At 6% hourly interruption, a ten-hour job has a 46% chance of being interrupted at least once. Half your runs die.
Tightly coupled multi-node training. One lost rank means a collective operation stalls, and depending on the framework the whole job either restarts or hangs until a timeout. The effective interruption rate multiplies by the number of nodes.
Anything with a latency SLO. Serving on spot means a fraction of your requests fail at exactly the moment your provider needs capacity back, which is correlated with when everyone else needs capacity back.
Part Four: The workloads that should always be on spot
Batch inference over a fixed corpus. Checkpoint at the shard boundary, which is free, and interruptions cost nothing because the unit of work is small.
Hyperparameter sweeps. Each trial is independent and short. A killed trial is a lost trial, not a lost job, and the scheduler just resubmits.
Data preprocessing. Idempotent, restartable, and the bottleneck is IO anyway.
The pattern: spot is for work whose unit of completion is small relative to the interruption interval. If a unit of work takes longer than the mean time between interruptions, spot is a bet against yourself.
Part Five: The hybrid that most teams should run
Reserve a floor that covers your steady-state load, and put the burst on spot with a fallback to on-demand when spot capacity is unavailable. The floor should be the capacity you use more than 70% of the time, not your peak.
Two operational details matter more than the split. First, tag every spot instance with the job and the checkpoint location, so a reaper can restart it automatically without a human reading logs. Second, track interruptions per thousand GPU-hours as a first-class metric. If it drifts above your model's assumption, the crossover moves and your reservation decision is stale.
The transferable lesson
Discounts are priced for the inconvenience they cause, and the inconvenience is variance. Any time a vendor sells you something cheaper in exchange for unreliability, the correct analysis is expected cost including recovery, not unit price.
The teams that run cheap GPU fleets are not the ones that found a better discount. They are the ones that made their work restartable in small units, which made the discount usable. Cheap capacity is not a purchasing decision, it is an engineering property of the workload, and you build it before you buy it.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)