DEV Community

Cover image for My Server Sleeps at Night. That Turned Out to Be a Security Feature.
Dhruv Malaviya
Dhruv Malaviya

Posted on Edited on

My Server Sleeps at Night. That Turned Out to Be a Security Feature.

I gave my side project a sleep schedule on Krova Cloud to save money — power off at 1 AM, wake at 7. The surprise side effect: zero attack surface while asleep. Here's the setup, the cron, and the trade-offs.

At 3 a.m., my production server is off. Not down — off. On purpose.

It started as a money thing. The project has human hours: traffic ~8:00 to midnight, dead until morning. The old VPS ran 24/7 and I paid for eight idle hours every night. Also, 3 a.m. on a VPS is when the botnet zoo shows up — brute-force loops, scanner sweeps, auth-log horror anthologies.

Then it clicked: this project has a sleep schedule. Servers are allowed to have sleep schedules.

The setup
The project now lives on a Cube on Krova Cloud — a Firecracker microVM with its own kernel and no public IP (private NAT'd network, managed TLS ingress, only explicitly opened ports reachable). The sleep schedule is the dumbest automation I own:

# crontab -e — on a host OTHER than web-1 (CI cron / small always-on box)
0 1 * * *   krova cubes power-off web-1   # lights out: compute billing stops, disk preserved
0 7 * * *   krova cubes wake web-1 && sleep 15 && curl -sf https://app.example.com/healthz || curl -sS -X POST "$ALERT_URL" -d 'text=web-1 failed to wake'
Enter fullscreen mode Exit fullscreen mode

Those are real CLI verbs (@krovacloud/cli), not pseudocode. Semantics per the docs:

  • running → billed per minute for compute + disk
  • stopped → billed for disk storage only
  • data survives both directions; wake starts the Cube from its saved disk

So sleeping 8 hours a day removes ~a third of compute time from the bill, and the stopped hours cost pocket change. On my 2 vCPU / 4 GB / 40 GB Cube that's the difference between "hosting" and "rounding error." (krova pricing prints the per-resource hourly rates if you want to do your own math.)

The surprise: the 3 a.m. noise died
First week I checked the logs out of habit. Nothing. No brute force, no scanners. Of course:

  • no SSH daemon listening
  • no web server process
  • nothing in memory, no kernel of mine executing
  • nothing to scan, because there's no public address anyway

You cannot exploit a process that isn't running. The nighttime attack surface isn't small; it's zero. Awake, the Cube still isn't a scannable box — no public IP, default-deny inbound — so the daytime surface stays tiny too.

Verify the states yourself:

krova get web-1 --json | jq -r .state   # "stopped" at 1:05 AM
krova cubes wake web-1
krova ssh web-1 -- uptime               # back in seconds, disk intact
Enter fullscreen mode Exit fullscreen mode

What sleep forced on my architecture (all good things)

  1. All state on disk. In-memory state dies at 1 a.m., so sessions in Redis-on-disk or DB rows, not globals. Sleep cured my lazy state management.
  2. Boot must be boring. Everything starts via systemd units; if wake + boot doesn't produce a working app with zero human input, that's a bug I fix in daylight.
  3. Health check = first request. I curl the domain after wake in the same cron pipeline; if boot ever fails, I know before users do.
0 7 * * *   krova cubes wake web-1 && sleep 15 && curl -sf https://app.example.com/healthz || curl -sS -X POST "$ALERT_URL" -d 'text=web-1 failed to wake'
Enter fullscreen mode Exit fullscreen mode

Where the schedule lives, and the two ways it dies

A commenter caught that v1 of this post never said where the crontab runs. Outside the Cube, as above — a powered-off host can't wake itself, and the 7 AM alert branch can't fire from inside a stopped Cube. But that creates the mirror failure: if that host dies, the Cube never sleeps, everything looks healthy, and the bill silently goes back to 24/7. So each failure gets a checker that is alive when it fires:

  • Asleep when it should be awake → the Cube is off, it can't report itself. Only an external check notices (the 7 AM healthz + alert above).
  • Awake when it should be asleep → the Cube is on, so something inside can talk:
# inside web-1 — this one MAY live on the Cube, it only needs to fire while awake
0 2 * * *   awake-during-sleep-window.sh   # alerts if the Cube finds itself running at 2 AM
Enter fullscreen mode Exit fullscreen mode
  • Any time — the interval watcher. Both bullets above are point samples; the bill is an interval. A stray wake at 3 AM slips between them and looks healthy by 7. The right observable turned out to be the state log Krova already emits: register a webhook for cube.running / cube.stopped (HMAC-signed, timestamped) and you get every transition:
POST /spaces/$SPACE/webhooks
{ "url": "https://watcher.example.com/krova",
  "events": ["cube.running", "cube.stopped"] }
Enter fullscreen mode Exit fullscreen mode

Alert on any cube.running outside the 7 AM window the second it happens; sum the running→stopped intervals nightly — ~960 compute-minutes expected, more is the alarm. The bill is the ground truth; the samples were approximations.

Three checkers now: two point samples for the cheap obvious failures, one event stream for the interval. And the schedule itself lives outside anything it controls.

Objections, fairly handled

  • Wake isn't instant. First morning request waits a few seconds for boot. Correct choice for side projects / internal tools / staging with quiet hours. Wrong choice for 24/7 global SaaS — don't sleep those.
  • Sleep ≠ security while awake. It shrinks the window; it doesn't replace the walls. The daytime Cube still needs its real properties: no public IP, own kernel, explicit ports only. Krova gives you those; sleep is just schedule math on top.
  • Timezones. If your users are global, your "quiet hours" may not exist. Look at actual traffic before copying my cron.

The reframe

We treat "always on" like a moral quality. For a lot of workloads it's just an expensive, attackable assumption. I added sleep to save money; the quietest security upgrade I've ever shipped was the side effect.

Give it a bed with no address. Let it sleep.

Does anyone else run real workloads on a sleep schedule? Or does powering off "production" at night terrify you? Which camp are you in, and what do your quiet hours look like? Comments open.

Top comments (5)

Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen

Worth saying where that crontab lives, because the two lines cannot both run on web-1. If the schedule sits on the Cube it powers off, the 1 AM line is the last thing that host ever executes: the 7 AM wake never fires, and the alert branch on that same line cannot fire either, because the process that would send it is off. That is the single failure your health check cannot report, since the checker is inside the thing being checked. Moving the schedule to a separate host fixes it and flips which direction goes quiet, though: if that host dies, the Cube just never sleeps, and nothing alarms while the bill goes back to 24/7.

Collapse
 
dhruv_malaviya_cdcc71e595 profile image
Dhruv Malaviya

The post was ambiguous about it. The crontab lives outside the Cube (CI cron), since a powered-off host can't wake itself, and as you said the alert branch can't fire from inside a stopped Cube either.

Your mirror failure made me update the post: if the scheduler host dies, the Cube never sleeps and nothing alarms. Now each failure has a watcher that's alive when it fires , the external 7 AM healthz + alert catches "asleep at noon," and a 2 AM cron inside the Cube catches "awake when I should be asleep." That second one may live on the Cube, since it only needs to fire while awake.

Thanks for the push , better post for it.

Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen

Both watchers are point samples, and the bill isn't. Something that wakes the Cube at 3 AM, a stray wake or a retried CI job, slips past the 2 AM check because that check already ran, and by 7 AM it is up and healthy, which is exactly what a good morning looks like. The interval observable is already in the pricing semantics you quoted: running bills per minute, stopped bills disk only, so compute-minutes per night is the number, and 24 where you expected 16 is the alarm. I haven't run Krova myself, so whether that shows up as a state log or a usage figure is yours to check.

Thread Thread
 
dhruv_malaviya_cdcc71e595 profile image
Dhruv Malaviya

Did the homework: it shows up as a state log, not a usage figure. The v1 API has no usage endpoint, but webhooks emit cube.running and cube.stopped , HMAC-signed, timestamped, 30 days of delivery history. That's the interval observable you described.

Post now has the third watcher: subscribe to those two events, alert on any out-of-window cube.running the moment it happens, and sum running→stopped intervals nightly for compute-minutes , 960 expected, 1440 is the alarm. Event-driven beats my 2 AM point sample outright; the samples stay as cheap belt-and-suspenders.

You've now shaped this post twice. If you ever run Krova, the first bill-vs-log diff is on me.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.