I built Ark to test whether backups from a production Ubuntu VPS could actually recover into AWS. The first recovery drill achieved a 1m 58s RTO using a real backup.

I Built a Disaster Recovery Tool Because Row Counts Lied to Me

A recovery plan that has never been executed is a hypothesis, not a plan.

I work with production infrastructure that isn't running entirely in AWS. A lot of it runs on Ubuntu VPS infrastructure.

That works well from a cost perspective, but it creates an obvious question:

What happens if the production environment disappears?

We have backups.

But having backups and being able to recover from them are two different things.

That's why I started building Ark, a small disaster recovery tool that uses AWS as a low-cost recovery environment for an Ubuntu VPS production system.

Then I ran an actual recovery drill.

That's where things got interesting.

The problem started with backup verification

I wanted a simple way to verify that a backup was actually usable.

At first, checking things like database row counts seemed reasonable.

So I tried it.

I destroyed both Docker volumes in a test environment, restored the application, and ran the seed process again.

The result looked fine:

51 users
200 repositories
400 actions

Everything matched.

If I only looked at the row counts, I would have said:

The restore worked.

But it didn't.

The Git object store was different.

The reason is that recreating the same logical dataset doesn't necessarily produce the same underlying data. Git commits contain metadata such as timestamps, so the resulting objects can be different even when the visible data looks the same.

That was the part that bothered me.

The verification was telling me everything was okay, while the actual data told a different story.

So I stopped treating row counts as proof that a backup was valid.

Building Ark

I wanted the recovery process to be something I could actually execute, not another document sitting somewhere.

The basic flow is:

Production Ubuntu VPS
        │
        ▼
      Backup
        │
        ▼
    S3 Storage
        │
        ▼
   AWS Pilot Light
        │
        ▼
  Recovery Drill
        │
        ▼
      Verify

Ark handles the backup, recovery, and verification process.

Backup

The backup process collects:

• database dumps
• Docker volumes
• configuration files

Each backup has a manifest containing SHA-256 checksums.

The backup is uploaded to S3, with lifecycle rules for older data.

One small decision here is important:

The manifest is uploaded last.

The actual backup files go first.

If something goes wrong halfway through the upload, there is no completed manifest telling the recovery process that the backup is ready.

So:

Backup files → upload
       ↓
Verification
       ↓
Manifest → upload last

No completed manifest means the backup isn't considered ready for recovery.

Recovery environment

Terraform manages the AWS side:

• VPC
• security groups
• launch template
• recovery configuration

I don't keep an EC2 instance running all the time.

The idea is closer to a pilot light: keep the infrastructure definition ready, then create the compute environment when a recovery drill or actual recovery is needed.

A fresh EC2 instance can then:

1. download the latest completed backup
2. verify the checksums
3. restore the database
4. restore the volumes
5. start the application
6. check whether the application is actually ready

That last part matters.

A successful restore command isn't enough.

I want the application to come back.

The first recovery drill

I ran Drill #001 on September 9, 2026.

The test launched a fresh EC2 instance, downloaded the latest completed backup from S3, restored the environment, and checked the result.

The numbers were:

RTO: 1m 58s     target < 15m     PASS
RPO: 182m       target < 6h      PASS

Provision EC2       0m 20s
Instance ready      0m 04s
Download backup     0m 02s
Restore             0m 25s
Verify              0m 12s

Database:           PASS
Volume integrity:   PASS
Application:        PASS

RESULT: PASS

The RPO was 182 minutes because that was the age of the latest completed backup when I ran the drill.

The RTO was 1 minute 58 seconds against a 15-minute target.

That's a good result.

But I don't want to claim that this means we have a two-minute disaster recovery system.

The test dataset was only around 80 MB.

What matters more to me is that the complete path actually worked:

Fresh EC2
   ↓
Download backup
   ↓
Restore
   ↓
Verify
   ↓
Application ready

And the recovery used a real backup, not a test fixture created specifically for the drill.

That's the part I wanted to prove.

What if the drill fails?

A recovery drill doesn't have to pass every time.

If it fails, that's useful too.

For example:

Drill #002

RTO: 18m 42s
Target: <15m

RESULT: FAIL

Failure:
Application readiness check timed out.

Action:
Investigate container startup dependency
and adjust the recovery process.

I'd rather discover that during a scheduled drill than during a real outage.

That's one of the main reasons I wanted the drill automated.

Running it manually once doesn't tell me much.

Running it repeatedly gives me evidence.

Why AWS Pilot Light?

The production environment doesn't need a second full-time copy just to have a recovery option.

For this use case, AWS can act as the recovery environment.

The infrastructure is defined with Terraform, while the EC2 compute only exists when I need it.

That keeps the ongoing cost low.

For the test environment, the first recovery drill cost about $0.01.

The point isn't that every production disaster recovery environment will cost one cent.

It won't.

The point is that testing the recovery path doesn't have to be expensive.

What I learned

The biggest lesson wasn't AWS.

It was that backup success and recovery success are different things.

A backup job can finish successfully.

A database restore can finish successfully.

The row counts can match.

And you can still have a recovery problem.

The only way I could really answer the question "Can we recover?" was to actually recover.

That's also why I don't see RTO and RPO as numbers that belong only in a document.

They're measurements.

If you have never run the recovery process, you don't really know what your RTO is.

If you don't know which backup you can actually restore, you don't really know your RPO either.

What's next for Ark?

There is still a lot I want to improve:

• Route 53 health-check-based failover
• Glacier cost calculation
• more automated test coverage
• more failure scenarios during recovery drills

Ark is open source under Apache 2.0.

GitHub: https://github.com/rendyachmad-dev/ark

I'm especially interested in how other engineers handle disaster recovery when production isn't running in AWS.

Because in the end, this is the question I wanted Ark to answer:

If production disappeared right now, could I actually bring it back?

A recovery plan that has never been executed is a hypothesis, not a plan.