Disclosure: I'm the founder of Kilawatt Cloud. I ran and timed every drill below on an internal test account, so this is our own test record and not independent reporting.
The question
Chip makers all sell the same promise: more compute, faster. For anyone buying GPUs through software, the next question matters more: what happens when a rented machine disappears halfway through a job?
For an agent that bought the job on its own, a failed job is a lost purchase and a reason not to buy again. I wanted measurements, so I destroyed ten live GPU machines mid-job and timed what the platform did next.
How the drills worked
Eight drills ran through the goose agent app on October 20. Each job was bought over x402 (USDC on Base) through our MCP server, and an operator destroyed the Vast.ai machine by hand after a health check passed. Two earlier drills ran on October 14, with a checkpointing job on a RunPod A100.
I measured two things:
Detection: the time from my kill to the platform marking the machine failed.
Failover claimed: the time from detection to the platform claiming a replacement on a different supplier.
The results
| Drill (Oct 20) | Detected (UTC) | Detection after kill | Failover claimed after detection |
|---|---|---|---|
| 1 | 04:34:02 | 184 to 243 s | 0.6 s |
| 2 | 04:38:03 | 124 to 183 s | 0.5 s |
| 3 | 05:10:03 | 184 to 243 s | 0.7 s |
| 4 | 05:28:02 | 184 to 243 s | 0.7 s |
| 5 | 05:30:03 | 184 to 243 s | 0.7 s |
| 6 | 05:36:01 | 122 to 181 s | 0.6 s |
| 7 | 05:46:01 | 182 to 241 s | 0.5 s |
| 8 | 05:54:01 | 183 to 242 s | 0.6 s |
On October 14, detection took 238 seconds in the first drill (RunPod to Hyperstack) and 121 seconds in the second (RunPod to Vast.ai). In the second, the job resumed automatically from its saved checkpoint 4, 12 seconds after the replacement was ready. It then went on to write checkpoints 5, 6 and 7.
Across all ten drills, the platform placed a replacement on a different supplier every time.
Why the ranges are wide
I recorded the minute I clicked Destroy, not the second, so each detection figure is a window of about a minute. Health checks also run every 2 minutes, which puts detection near 3 to 4 minutes after a kill.
What these drills do not show
Replacement-ready time. The database wasn't recording it. Replacements passed later health checks, but I have no timing, so I'm not claiming a recovery time for the October 20 drills.
Resume on the October 20 jobs. Jobs bought through x402 don't save checkpoints, so those eight measure detection and replacement only. Resume was shown only on October 14.
A median or worst case. Ten drills are too few, so I'm giving exact counts and ranges.
A supplier-removal drill. Not run yet.
Fully autonomous buying. A person typed each buy prompt and destroyed each machine. This shows the pay-per-job path and failover working together.
The 6 jobs I excluded
I paid for 14 jobs on October 20 and counted 8:
Four never booted on one Vast.ai host. They were auto-refunded, and I blocked that host in code.
One 600-second window was too short.
One was a replacement stopped at the end of its window, which isn't a drill.
What the drills broke
The endpoint said "rate limiting requests" for several different refusals. It should name the real limit and when it clears.
A job's concurrency slot stayed held until its 15-minute window ended, even after the instance was destroyed.
The never-ready refund didn't fire on time for two jobs.
Some hosts stall pulling the Docker image. Pre-caching it is on the list.
What's next
Detection is slower than I want at about 2 to 4 minutes, and shortening it is first on the roadmap. After that:
Record when each replacement is ready.
Run a larger set with median and slowest times.
Run a supplier-removal drill and publish it.
The full case study, with the method and the excluded attempts, is here: https://www.kilawattcloud.dev/super-intelligence
About the author: Damian Dixon is the founder of Kilawatt Cloud, based in Los Angeles.
Top comments (4)
Thanks for publishing the excluded attempts too, that's rare. The numbers make one thing clear: detection (~2–4 minutes after the kill) dominates, while the replacement claim is sub-second. Is detection heartbeat-based? Shortening that interval, plus frequent checkpoints, looks like where most of the remaining recovery time is.
The detection ranges carry more information than "health checks every 2 minutes". If one failed check marked a machine down, a kill at a random moment would be caught 0 to 120 s later. Your windows are 122 to 243 s, almost exactly 120 to 240, which is what you get when it takes two consecutive failed checks: the wait to the next check plus one more full interval. The detected timestamps fit too, all landing within 3 seconds of an even minute (04:34:02, 05:10:03, 05:54:01).
If that is the rule, interval and failure count are the two knobs: 30 s checks with two misses would put detection near 30 to 60 s, at the price of more false failovers on a supplier that drops one probe. Is the two-miss rule deliberate, and do you have the false-failover rate on those suppliers from normal operation?
Good read of the timestamps, and yes, that is deliberate. Health checks run every 2 minutes, and failover on a running job needs two consecutive "gone" readings more than 2 minutes apart, so detection lands in a window of roughly 2 to 4 minutes. A single failed check can't trigger it. We fixed a gap here on Sep 27, when a single transient error could trigger failover, and the incident review is on our public status page. Your 30-second example is the right trade-off: faster detection, but a higher risk of failing over a healthy machine.
On false failovers, we don't have a measured rate, so I won't quote one. What I can say is that every detection in our records falls inside a drill we ran on Oct 14 or Oct 20. We have not seen one during normal operation. That is a small sample with mostly short test jobs, so it says little about long runs. Shortening detection is the first item on our roadmap, and measuring false failovers properly is part of that work. For a first version, would you tune toward faster detection or fewer false failovers?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.