DEV Community

goodpa
goodpa

Posted on

The Escrow Layer — Or, "It's in the Repository" Is Not the Same as "It Can Be Rebuilt"

Today a hosting provider forced a password reset across its customer base (111 points). Somewhere in that crowd is a team that did everything right: credentials documented, secrets in a vault, an escrow copy of the source in a second location. They still spent their morning locked out of their own infrastructure, because the reset wasn't theirs to schedule.

That's the escrow layer. Not whether the box exists — whether the box is worth opening.

Last week's piece was about succession: who inherits the obligation. This one is about what you hand them. It turns out most escrow arrangements are a box of souvenirs.

The zip file fallacy

Ask a team what's in their software escrow and you'll usually get: "the repo." Sometimes a tarball. Occasionally a USB drive in a drawer with a label in someone's handwriting.

That's source code. Source code is the least useful thing in the box, because source code alone is a claim that a system can be built. It isn't a build.

Consider what actually has to be true on the day someone opens the box:

  1. The source builds. Not "compiles eventually after a week of archaeology" — builds, from a clean checkout, on a machine that exists today. If your build depends on a pinned toolchain that hasn't been updated in four years, the box contains a museum piece.
  2. The build environment is reconstructible. Base images, compiler versions, system libraries, the one native extension that only compiles with an old glibc. If this lives in someone's head or a laptop's shell history, it isn't escrowed.
  3. The credentials can be rotated. This is the one almost everyone misses, and today's forced reset is the proof. A box full of API keys is not an asset; it's a liability with an expiry date you don't control. The thing worth escrowing is the process for issuing new credentials — who can authorize it, what the recovery path is, which accounts are load-bearing.
  4. Someone knows which 20% matters. Every system has a core that carries the product and a periphery that carries the admin console. A successor who can't tell them apart will spend their first month on the wrong half.

Fail any one of those and you don't have continuity. You have a folder.

Three ways the box turns out empty

Today's front page supplied one example of each.

The keys expired without you. Hosting provider forces a global password reset. Nobody breached anything, nobody was at fault — a security event at the provider invalidated credentials at scale. If your continuity plan was "we have the keys in escrow," you just learned that the keys were never the asset. Access is a relationship with a counterparty who can revoke it unilaterally and on their own timeline. Escrow of a credential is escrow of a permission slip, not a capability.

What you escrowed is a moving target. OpenAI and Anthropic reportedly cut LLM prices by up to 80% in a few months (120 points). That's good news for buyers and terrible news for anyone whose continuity plan is "fall back to provider X at the price we negotiated." The thing being escrowed — access to a specific model, at a specific rate, with a specific behavior — is precisely the part the vendor changes without asking. You can escrow weights and you can escrow a contract. You cannot escrow a price.

The box has root in it. A root shell exploit on a consumer TV (763 points) is not a story about a TV. It's a story about every device you shipped whose firmware you no longer control and whose update channel you no longer own. If the thing in your escrow is a binary that runs with full privilege on hardware in someone's living room, the successor doesn't inherit an asset. They inherit an incident.

The four-item test

You can run this on a Tuesday afternoon. For each item, the answer is PASS, FAIL, or UNKNOWN — and UNKNOWN counts as FAIL, because the day you need it is not the day to discover it was a guess.

Item PASS looks like Usual reality
Source builds clean A fresh machine, a documented command, a green build, timestamped this year Builds on one engineer's laptop
Environment reconstructible Pinned base images + a script that provisions from zero "We use the same Docker image as prod"
Rotation path exists A named human who can issue new credentials, with a tested recovery route Keys in a vault, no rotation drill ever run
Core is documented One page listing the services that carry revenue and why Architecture diagram from 2022

Two of four is the common score. Two of four is not continuity — it's the beginning of continuity, and only if the two missing ones are the ones nobody needs. They never are.

What you actually owe your customers

Not a document. Three sentences, written down, that a customer could read without needing a translator:

  1. If our primary provider disappears tomorrow, here is what changes about the service you receive. Be specific about degradation, not about survival.
  2. Here is how long we can run degraded. A number. Hours or days, not "minimal impact."
  3. Here is who inside this company is accountable for that answer. A role, not a team name.

If you can't write the first sentence, you don't know your own dependency graph. If you can't write the second, you've never tested it. If you can't write the third, the answer is "nobody," and your customer is about to find out at the worst possible moment.

Six moves, in order of cost

  1. Run the build from a clean checkout this week. Not a fresh branch — a fresh machine. Note every undocumented step. That list is your real escrow gap.
  2. Move the rotation path out of people's heads. Written steps, named authorizer, one tested execution per year. Today's forced reset is the argument you can now make to your boss without inventing anything.
  3. Stop escrowing keys. Escrow the ability to get new ones. A vault with no rotation procedure is a drawer with better marketing.
  4. Write down the 20%. Which services carry revenue, which carry admin, and which exist because someone needed them in 2021. A successor with this page saves a month.
  5. Keep a copy of the build environment, not just the source. Base image digests, toolchain versions, the ugly native build steps. This is the part that's always missing and always the blocker.
  6. Test with a stranger. Hand the box to someone who has never touched the system, give them a day, and watch where they stop. That stopping point is the honest maturity level of your escrow.

The part nobody wants to hear

An escrow you have never opened is a hypothesis, not a plan. The forced password reset today was, for a few hundred teams, the first real test of a continuity assumption they'd been carrying for years — and the ones who passed weren't the ones with the most complete documentation. They were the ones who had done it once before, on a quiet day, when nothing was on fire.

The next piece in this series is about that: the restore drill. Why an escrow that has never been restored is indistinguishable from an escrow that doesn't exist, and how to run the drill cheaply enough that you'll actually do it twice a year.


Top comments (0)