I had a GitHub Actions workflow called Deploy to DigitalOcean. It had been sitting in the repo for weeks, fully wired up — SSH action, secrets references, the works. And yet every single time I shipped a backend change, I still had to SSH into the droplet and run git pull by hand.
I hadn't stopped to ask why until I actually went looking. It turned out the pipeline wasn't broken in any one obvious way — it was broken in six small, unrelated ways, stacked on top of each other, each one hiding the next. This is the story of digging through all of them, in the order I actually found them.
Act 1: The Deploy Job That Never Ran
The setup looked correct on paper. deploy.yml was gated like this:
on:
workflow_run:
workflows: ['CI']
branches: ['main']
types: [completed]
jobs:
deploy:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
Reasonable: only deploy if CI passed. So the first thing worth checking wasn't the deploy job at all — it was whether CI had ever passed on main.
gh run list --workflow=ci.yml --limit 5
Every single row: failure. Every deploy run: skipped. The deploy pipeline wasn't broken — it was doing exactly what it was told, gating correctly on a CI workflow that had never once gone green on main. I hadn't connected those dots because the failures and the missing deploys lived in two different tabs of the Actions UI.
Act 2: Two Bugs Wearing One Trenchcoat
gh run view <run-id> showed two jobs failing for completely unrelated reasons.
The linter job died immediately:
The specified python version file at: .python-version doesn't exist.
.python-version was sitting right there in the repo root. Except it wasn't — it was in .gitignore, left over from a pyenv boilerplate line nobody had questioned. actions/setup-python's python-version-file input needs the file to actually exist in the checkout, and a gitignored file never reaches CI no matter how real it looks on disk locally.
The frontend job died differently:
Error: [vitest-pool]: Failed to start forks worker for test files ...
Caused by: TypeError: webidl.util.markAsUncloneable is not a function
That error has nothing to do with my test code. ci.yml pinned node-version: '20', but jsdom@30 (a Vitest dependency) requires:
"engines": { "node": "^22.22.2 || ^24.15.0 || >=26.0.0" }
Node 20 doesn't have markAsUncloneable — jsdom needs it. Every frontend test run was crashing before a single test file even executed. The actual backend test suite (pytest) had been passing this entire time; it was just outvoted by two infrastructure bugs with nothing to do with the code being tested.
Fix: un-ignore .python-version, commit it. Bump node-version to '22'. Two-line diff, two root causes eliminated.
Act 3: The Debt That Was Always There
Pushed the fix, watched CI run again. pytest and frontend went green. linter failed again — different reason this time:
black....................................................................Failed
- hook id: black
reformatted job_board/jobs/services.py
isort....................................................................Failed
- hook id: isort
Fixing job_board/jobs/services.py
djLint formatting for Django.............................................Failed
- hook id: djlint-reformat-django
1 file was updated.
Formatting drift had been quietly accumulating in three files — a multi-line import that black wanted split, a .envs file missing a trailing newline, an index.html <meta> tag that djLint wanted wrapped. None of it mattered locally because nobody had pre-commit wired into their git hooks. It only mattered in CI, which is exactly where it had been failing, unnoticed, this whole time.
The fix here was almost mechanical: CI's own hook output is the diff. Applied it verbatim rather than re-deriving it — no ambiguity about what "correctly formatted" means when the formatter already told you.
Act 4: The Flaky Cache and the Duplicate Runs
Two more small papercuts surfaced once real CI runs were flowing:
A cache backend race. PR-triggered builds started intermittently failing with:
Error: cannot parse bake definitions: ERROR: failed to solve:
failed to load cache key: repository does not contain ref refs/pull/166/merge
This is a known docker/bake-action + GitHub Actions cache quirk: a PR's merge ref gets recomputed (or invalidated) if the PR updates or merges while the cache backend is still resolving it. It never affected the push-triggered run on main — the one that actually gates deploy — but it left a false red X on every promotion PR. Buildx's GHA cache backend has a purpose-built escape hatch for exactly this:
django.cache-from=type=gha,scope=django-cached-tests,ignore-error=true
ignore-error=true degrades a cache-restore failure to a cache miss instead of failing the whole job.
Duplicate CI runs. With both pull_request: branches: [main] and push: branches: [main] as triggers, every promotion ran the exact same commit through CI twice — once when the PR opened, again the instant it merged. Since deploy only ever cares about the post-merge state of main, the fix was to drop the pull_request trigger entirely and let push do the only job that mattered.
While in there, I also pulled the frontend lint/test job out of backend CI altogether — the frontend deploys through Vercel, which already runs its own build/lint pipeline as a separate PR check. There was no reason a frontend typo should ever gate a DigitalOcean backend deploy, so frontend/** also got added to paths-ignore — a frontend-only commit no longer triggers a backend rebuild at all.
Act 5: CI Was Finally Green. Deploy Still Failed.
For the first time, CI showed success on a push to main. And for the first time, Deploy to DigitalOcean actually ran instead of showing skipped. It failed in 12 seconds:
ssh.ParsePrivateKey: ssh: no key found
ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
gh secret list --repo <org>/<repo>
Zero rows. DO_HOST, DO_USERNAME, DO_SSH_KEY, DO_PORT — none of them existed. The workflow had been referencing four secrets that had simply never been created. CI passing had done its job; it just exposed the next layer down.
Act 6: Two SSH Gotchas
Setting these up surfaced two mistakes worth writing down because they're both easy to make and easy to misdiagnose.
Gotcha 1 — ssh-copy-id from inside the box. Trying to install a public key onto the droplet while already SSH'd into that droplet doesn't work: ssh-copy-id opens a new outbound connection back to the same host to install the key, but that connection has no valid key to authenticate with yet. Classic chicken-and-egg. Once you're already sitting in a shell on the target machine, skip ssh-copy-id and just append directly:
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
Gotcha 2 — mangled secret content. Even after the public key was trusted server-side, the deploy still failed with the exact same ssh: no key found error. The private key had been set as a GitHub secret via a string interpolation (--body "...") rather than a direct file redirect, which had silently collapsed the multi-line PEM block. The fix is to never let anything touch the formatting:
gh secret set DO_SSH_KEY --repo <org>/<repo> < ~/.ssh/id_ed25519
Redirecting the raw file guarantees byte-for-byte fidelity, including the -----BEGIN/END----- lines and every internal newline that a shell string would otherwise eat.
Act 7: Right Key, Wrong Directory
With auth finally working, the deploy script actually reached the droplet and ran — into a new error:
err: bash: line 1: cd: /home/***/apps/gigglegigs: No such file or directory
err: fatal: not a git repository (or any of the parent directories): .git
err: open /home/***/production.yml: no such file or directory
deploy.yml assumed the checkout lived at ~/apps/gigglegigs. It didn't — a quick ls ~ on the droplet showed the actual clone sitting at ~/GiggleGigs. That path had apparently been copy-pasted from a different project's deploy config and never actually validated, because the workflow had never gotten far enough to hit it before. One-line fix, and the deploy script finally ran start to finish: build, collectstatic, migrate, up -d, all green.
Act 8: The First Real Deploy Broke the Site
Two minutes after that first fully green deploy, the admin's user list started returning 502 Bad Gateway. The instinct is to panic here — first automated deploy, and the site's down? But the logs told a much less dramatic story:
django-1 | PostgreSQL is available
django-1 | 186 static files copied to '/app/staticfiles', 538 post-processed.
django-1 | [2026-09-01 12:48:17 +0000] [1] [INFO] Starting gunicorn 21.2.0
django-1 | [2026-09-01 12:48:17 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000
Gunicorn didn't finish booting — Postgres wait, collectstatic, worker fork — until 12:48:17. The 502 was timestamped 12:48:16. One second earlier. Nothing crashed; I'd just refreshed the admin during the exact window where docker compose up -d had already killed the old django container and the new one wasn't listening yet.
That's not really a bug in the app — it's a gap in the deploy setup. Traefik here uses the static file provider, routing unconditionally to http://django:5000 with zero awareness of container readiness:
services:
django:
loadBalancer:
servers:
- url: http://django:5000
Every deploy, without exception, would keep hitting this same window. The right fix is an active health check so Traefik stops sending traffic to a container until it's actually answering:
services:
django:
loadBalancer:
servers:
- url: http://django:5000
healthCheck:
path: /healthz/
hostname: api.example.com
interval: '5s'
timeout: '3s'
Two things worth calling out about that config. First, /healthz/ didn't exist — it needed to be a genuinely trivial, unauthenticated view (no DB touch, no auth, just 200 OK), because coupling infra health to a real business endpoint means a permissions change or a slow query silently takes your health check down with it. Second, hostname isn't optional: without it, Traefik's health check request sends Host: django (the internal service name), and ALLOWED_HOSTS — quite correctly — rejects that as a DisallowedHost, returning 400 and making Traefik think the container is permanently unhealthy. Setting hostname to the real public domain makes the health check's request indistinguishable from real traffic, from Django's point of view.
While already in production.yml, I added restart: unless-stopped to every long-running service (everything except the one-off backup container). Without it, a droplet reboot — a DigitalOcean maintenance window, a crash, anything — leaves every container down until someone notices and SSHes in to bring them back up manually. unless-stopped means Docker brings them back on its own the moment the daemon starts.
What Actually Was Wrong, All Together
Laid out in one place, the full list is almost funny:
- A gitignored file broke Python setup in CI
- A pinned Node version was two majors behind what a dependency required
- Formatting drift had been silently failing lint for who knows how long
- A GHA cache backend race gave false negatives on PR checks
- CI ran twice per promotion for no reason
- Zero deploy secrets existed
- A private key got mangled by shell interpolation
- The deploy script's target directory was simply wrong
- Traefik had no way to know when a container wasn't ready yet
- Nothing would restart itself after a reboot
No single one of these was hard to fix once found. What made this a slog was that each bug hid the next one — CI had to go green before deploy secrets could even matter; deploy secrets had to work before the wrong directory path could surface; the deploy had to actually succeed before the Traefik timing gap became visible at all. A pipeline with ten small breakages in series doesn't look like ten breakages. It looks like one big wall, and the only way through it is to fix the outermost failure, rerun, and see what the next layer reveals.
Takeaway
If your "automated" deploy still requires someone to manually pull and rebuild, don't assume the deploy job itself is broken — check whether the thing gating it has ever actually succeeded. A pipeline that's silently and permanently gated behind a red CI run looks, from a distance, exactly like a pipeline that doesn't exist. The fix isn't usually one big rewrite. It's peeling one failing layer at a time until the thing underneath finally gets a chance to run — and then trusting the next error message, however unrelated it looks, to point at the next real problem.
Top comments (8)
Act 8 is the one I'd push on, because the health check you landed on solves half the problem and it's worth knowing which half.
What you've built there is a liveness check — trivial view, no DB, no auth, just 200. Right call for the failure you hit, and the hostname point is the one everybody loses a day to. But a container can answer that endpoint perfectly while its connection pool is still empty or its queue consumer hasn't attached. Traefik starts routing, requests land, and instead of a clean 502 for one second you get real 500s from a container that reports healthy. Liveness answers "is the process up". Readiness answers "can it serve a request end to end". You usually want both, on separate paths, with only readiness wired into the load balancer.
The catch is that readiness is a dependency graph you're committing to. Touch a downstream service in it and you've coupled your rotation to someone else's uptime — one flaky dependency and the whole fleet marks itself unready at the same time. So it should check only what that container needs to serve its own traffic, which is a design decision rather than a config line.
Other half is docker compose up -d itself. Health checks shrink the window, they don't close it, because compose still stops the old container before the new one is ready. Getting that to zero needs the new instance passing readiness before the old one goes away.
The "each bug hid the next one" framing is the real lesson though. Ten breakages in series don't look like ten breakages.
Agreed on all three points, and the liveness/readiness distinction is the right vocabulary for what I hand-waved as "actually answering." For this stack, readiness would be roughly: SELECT 1 against Postgres and nothing else — Redis/Celery being down degrades background jobs but shouldn't pull the web container out of rotation, which is exactly the scoping decision you're describing. That's the version I'd wire into Traefik's healthCheck, with the bare 200 kept on a separate path for restart policies.
On the compose gap — yes, up -d is stop-then-start, so the window shrinks to the new container's boot time but never hits zero. The honest options at this scale are: live with a health-gated few seconds, script a start-first rollout (scale=2, wait for the new one to go healthy, kill the old — the docker-rollout plugin does exactly this), or admit you want Swarm's order: start-first and stop pretending compose is an orchestrator. For a solo droplet I'll probably take option one with the readiness fix, since the remaining window is now "gunicorn fork time" rather than "gunicorn + collectstatic + migrations" — but you're right that I solved half the problem and should say so.
This might be an Act 9.
Scoping readiness to
SELECT 1and leaving Redis out is the right call, and it's the half most people get backwards.One trap on that specific check though.
SELECT 1proves the pool can hand you a connection. It doesn't prove the pool isn't nearly empty. If readiness borrows from the same pool the app serves from, then under load it either succeeds by taking the last free connection, or it fails and pulls the container out of rotation at the exact moment it's busiest. That second one is a feedback loop — the container gets removed, its traffic moves to its neighbours, their pools saturate, they fail readiness too. A check meant to protect you takes the fleet down instead. Cheap fix is a dedicated connection outside the app pool, or just accept that this check answers "is Postgres reachable" and not "can I serve".On Act 9, the thing I never see written is that start-first doesn't get you to zero either. It closes the window where nothing is listening and opens a different one, where the old container takes SIGTERM with requests still in flight. Traefik needs a moment to notice the old server is gone, and gunicorn needs
graceful-timeoutlong enough for the request to actually finish. Start-first without a drain window just moves the dropped requests from the front of the deploy to the back.For a solo droplet I'd land exactly where you did. Health-gated compose, remaining window is fork time, done. The rollout plugin earns its keep when a deploy costs you money, not before.
The cascade scenario is the part I hadn't thought through — a readiness check that competes with traffic for the resource it's supposed to be measuring is an observer effect with teeth. For gunicorn sync workers the practical version of "dedicated connection" is probably just psycopg.connect() fresh inside the health view rather than going through Django's ORM/pool at all — a connect-and-close per check is cheap at a 5s interval, and it cleanly answers "is Postgres reachable from this container" without ever touching the pool the workers serve from. Which, per your framing, is the honest scope of the check anyway.
The SIGTERM point closes the loop on the whole thread nicely: the failure just migrates. Nothing-listening window → boot window → drain window. Each fix relocates the dropped requests rather than eliminating them, until you've explicitly handled both ends — readiness gating the front, graceful-timeout + LB deregistration lag covering the back. Written down like that, "zero downtime" is less a config line and more a small distributed-systems contract.
And agreed on the plugin threshold. When a dropped request costs an apology instead of revenue, health-gated compose is the right amount of engineering. Thanks for pressure-testing this all the way down — this exchange is better than the article.
The Act 2 rule is one step wider than what git actually does, and the gap runs in the direction that costs a day. I built both branches on 2.50.1 (Apple Git-155): a
.python-versionthat was ignored before it was ever added is missing from a fresh clone, which is your case, but one that was committed and then added to.gitignorestays in the index and does show up in the clone. So.gitignoremembership is not what decides it, the index is, and both branches look identical on disk locally - which is why "it looks real on disk" cannot be the discriminator either way.The one-command version is
git ls-files --error-unmatch .python-version, which exits nonzero in exactly the case wheresetup-pythonis about to fail on a missing file. Cheaper than reading.gitignore, and it stays correct after someone re-adds that ignore line for a file that is already tracked.You're right, and this is a sharper mental model than the one I wrote. "Gitignored" was shorthand for what mattered in my case (never tracked), but the index is the actual source of truth — a tracked-then-ignored file ships fine, and both states are indistinguishable from ls. That's exactly why I burned time on this: the file looked present because it was present, locally.
git ls-files --error-unmatch is going into my debugging toolkit — it's also a nice one-liner to drop into a CI step or pre-commit check for any file that setup actions depend on (.python-version, .nvmrc, etc.), so the failure message says "not tracked" instead of "doesn't exist." Thanks for actually testing both branches instead of theorizing.
Your step‑by‑step walkthrough of the misleading CI/CD behavior makes the debugging process crystal clear and easy to follow. I think this guide would be a valuable resource for the community on ZyVOP (zyvop.com) if you ever syndicate or cross‑post it there.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.