Your backup job ran successfully last night.
This morning, a migration deleted the wrong data.
You open the backup dashboard. Everything is green.
Then someone asks:
“How long until the app works again?”
That’s when you discover whether you have a recovery process—or just backup files.
A successful backup is important. Restoring it and checking the application gives you much stronger evidence that you can recover.
Here’s a practical drill you can run before an incident makes it urgent.
1. Define what you need to recover
Start with a specific scenario:
“Our main database is unavailable. We need to restore the application using an existing backup.”
Write down two targets:
Recovery Time Objective (RTO): How long can the service be unavailable?
Recovery Point Objective (RPO): How much recent data can you afford to lose, measured in time?
For example:
Recovery time target: 60 minutes
Acceptable data-loss window: 15 minutes
These are examples, not universal recommendations. Choose targets based on what your users and business can tolerate.
AWS recommends performing recovery tests to validate whether your backup process meets these objectives.
2. Restore an existing backup into an isolated environment
Use a backup produced by your normal process. Creating a special export for the drill can hide problems in the process you actually depend on.
Restore into a separate database or environment with a clearly different name:
app_restore_drill
Check the destination before running restore commands.
Also isolate the recovered application from production integrations. You don’t want a test worker sending real invoices, emails, or payment requests.
Protect restored customer data with appropriate access controls, and remove the test environment when the exercise is complete.
Google Cloud’s recovery guidance recommends testing backups in a non-production environment.
3. Check whether you have everything the restore needs
Finding the backup file is only the beginning.
Can you access:
- The backup storage account?
- Required decryption keys?
- Compatible database tooling?
- Necessary extensions?
- Database roles and permissions?
- Application configuration?
For PostgreSQL, a single-database pg_dump does not include cluster-wide objects such as roles and tablespaces. Those need a separate recovery plan; PostgreSQL provides pg_dumpall for global objects.
This is exactly the kind of missing dependency a drill should reveal.
4. Inspect the restored data
A completed restore command doesn’t prove that you recovered the right data.
Check important business records:
- Can you find a known customer?
- Are their orders or invoices present?
- Do payment totals match the expected state?
- Are relationships between records intact?
- Are permissions behaving correctly?
Compare against the backup’s expected recovery point. A backup from midnight won’t contain an order created at 9 a.m.
Use known transactions or recovery markers where possible. A single timestamp or total row count can miss problems.
Verify the data your application depends on, not merely the existence of its tables.
5. Start the application against the restored database
Now test the recovered application.
Can a test user sign in?
Can they open an existing record?
Can they create and update a new record?
Can the application generate an important report?
This can expose problems that database checks alone miss:
- Missing permissions.
- Unavailable extensions.
- An incompatible application version.
- Incorrect connection settings.
- Missing files referenced by database records.
Your database might contain a document’s filename while the actual document lives in object storage.
Recovering the database is one part of recovering the service.
Check the other stateful dependencies your application needs.
6. Measure the whole recovery
Start the timer when you begin the recovery procedure.
Include time spent:
- Finding and accessing the backup.
- Preparing the environment.
- Restoring data.
- Configuring the application.
- Validating important workflows.
Stop when the service reaches your defined recovery criteria.
Record which incident stages the drill did not simulate, such as detection, escalation, or production traffic cutover.
You might discover:
Database restore: 12 minutes
Environment and application setup: 25 minutes
Validation: 18 minutes
Total drill time: 55 minutes
Reporting only the 12-minute restore would hide most of the work.
7. Measure the data-loss window honestly
Suppose an incident happens at 10 a.m., but the latest recoverable state is midnight.
That leaves a ten-hour gap.
A successful restore doesn’t make that gap acceptable.
If your target is 15 minutes, your backup strategy needs to support a sufficiently recent recovery point.
Depending on the database and platform, that may involve more frequent backups, transaction-log archiving, or point-in-time recovery.
Also distinguish the failure you’re testing:
- If the database disappears, you want a recent recoverable state.
- If a bad change corrupts data, you need a recoverable state before that change.
Newer isn’t always the correct recovery point.
8. Write instructions another person can follow
Document the process while you perform it.
Include:
- Where backups are stored.
- How authorized people obtain access.
- Which tools and versions are required.
- How to select the recovery point.
- How to identify the restore destination.
- How to validate the result.
- Who makes the decision to return the service to users.
Keep credentials in your approved secret-management system, rather than copying them into the document.
Then have another team member follow the instructions.
If they need to ask you about several undocumented steps, you’ve found useful improvements.
A small recovery report
Your first report can be simple:
Scenario:
Main database unavailable.
Backup used:
Identifier and expected recovery point.
Restore destination:
Isolated drill environment.
Validation:
Sign-in, record lookup, create/update, reporting.
Measured recovery time:
55 minutes.
Observed recovery point:
Recorded and compared with the data-loss target.
Problems:
Missing role setup and outdated configuration instructions.
Follow-up:
Automate role provisioning and update the runbook.
Repeat the drill after fixing the gaps. Schedule future exercises according to the importance of the service and changes to its infrastructure.
Before your next deployment
Ask your team:
“When did we last restore a backup and run the application against it?”
If the answer is “never,” you’ve found a valuable engineering task.
Start small: restore one existing backup, validate one important workflow, and record what happened.
The first drill doesn’t need to be perfect. It needs to reveal what would otherwise surprise you during an incident.
Have you ever restored a backup that looked healthy but wasn’t enough to recover the application?
Further reading
Top comments (9)
One thing I'd add to the recovery drill is verifying that indexes and other database metadata are restored correctly. I've encountered several situations where the data came back successfully, but indexes were missing or outdated. The behavior can vary considerably depending on the database vendor, backup mechanism, and restore process.
A basic smoke test, such as checking whether record X exists, might pass without revealing the problem. Missing conventional indexes can cause serious performance degradation under realistic workloads, while missing or incorrectly restored vector search indexes can break retrieval in AI applications even when all the underlying documents and embeddings are present.
It's another reason to test representative application workloads after a restore, not just whether individual records survived. Successfully restoring the data and successfully recovering the system are two different things.
Section 3 hides a trap worth calling out. When a drill restore stops on
role "app_readonly" does not exist, the quick way past it ispg_restore --no-owner --no-acl, and then the drill goes green while a real recovery would still be missing every grant. I'd keep apg_dumpall --globals-onlyfile next to each dump and restore it first, so the drill fails for the right reason. For step 5, I'd also give the restored app its own environment name in the error tracker, likerestore-drill, so the errors it throws stay visible without paging whoever is on call.Love the idea of getting someone else to follow the steps too. If only the person who wrote it can follow it, that’s a problem
The pg_dump not including roles bit is exactly the kind of thing you only learn during a real incident. Step 5 is the one I'd push harder on, a restore that "worked" but the app can't log in is still a failed restore. For a solo dev with no staging, would you still do the full drill, or just restore into a throwaway local db and check a few known records?
For the bad-migration scenario, I'd seed three markers in the drill: a valid write before the migration, the destructive change, and a valid write afterward. Then restore to before the migration and check which markers survived. That makes the tradeoff visible: removing the corruption can also remove legitimate later work. I'd put that later-write reconciliation in the recovery report, so a successful restore isn't mistaken for recovering every valid transaction.
One thing I'd add to this is testing what happens after a point-in-time recovery, not just whether the database comes back.
Take PostgreSQL as an example. Suppose a bad migration drops a table at 10:15 AM. You restore to 10:14 using WAL archiving and promote the recovered instance. That recovery creates a new timeline.
Now imagine you need to investigate another issue and recover something from the original timeline. If your WAL archive retention or timeline history files aren't handled properly, you might have a much harder recovery problem than expected.
This is why I'd include multiple recovery scenarios in the drill: accidental deletion, corrupted data, and recovery after a previous PITR. A restore that works once doesn't necessarily prove that your recovery chain is reliable.
The green dashboard line is too real. The part I'd actually copy is restoring into an isolated env, because a test worker sending real invoices is exactly the mistake I'd make in a drill. When you ran yours, what was the missing piece — keys, roles, extensions — or did it just work first time?
the pg_dump not including roles and tablespaces point is the one that bites people. we ran a restore drill last year and the dump came back fine but every grant was missing, so the app booted and then threw permission errors on half the routes. now the drill script also runs a smoke query per role. do you time the drill end to end and compare it to the RTO number, or just check that it finishes?
the green dashboard is usually a size check against yesterday, and slow shrinkage
never trips it. worse: once more than half your recent runs are truncated, the
median you compare against is itself a broken backup, so it starts voting for the
failure. a hand-written floor ("below this is impossible") catches the collapse;
only comparing runs catches the drift.