Today was the first troubleshooting task of the run rather than a build task, and it needs the opposite instinct. Build tasks reward moving. Repair tasks punish it. Change nothing until you know the cause.
What made it satisfying is that every system involved told me exactly what was wrong, in its own vocabulary, before I touched anything.
One Docker task, one AWS task. Configure Apache inside a running container with docker exec, then find out why an EC2 instance running nginx is unreachable. The tasks come from the KodeKloud Engineer platform.
docker exec, and why your shell leaving does not kill anything
docker exec -it kkloud bash
exec starts an additional process inside a container that is already running. It does not restart the container and it does not disturb PID 1. A stopped container has nothing to exec into, so docker start comes first.
-i and -t do different jobs and you need both. -i keeps stdin connected so the process can read what you type; -t allocates a pseudo-terminal so prompts and line editing behave. docker exec kkloud bash without them exits immediately with no output, which looks like a broken container and is not.
Inside, the first surprise for anyone used to a normal machine:
service apache2 restart # works
systemctl restart apache2 # System has not been booted with systemd as init system
Most images have no init system and nothing resembling systemd as PID 1. The SysV service wrapper still works because it calls the init script directly. That error message is precise and still manages to look like a broken installation.
Then the thing that surprises people in the other direction. Typing exit ends your shell and the container keeps running:
exit
docker ps # still there
A container's life is tied to PID 1, not to your session. This is the difference from docker attach, which connects you to PID 1 itself. Docker documents Ctrl+C there as terminating the container when it was started without -i and -t. exec gives you a second process; attach gives you the first one.
Two config files had to agree, which is a theme that comes back on Day 41. ports.conf decides what Apache listens on, and the virtual host block decides what it answers for. Change one and not the other and you get a server bound correctly that serves nothing.
And everything done this way lives in the container's writable layer. It vanishes with docker rm, and a second container from the same image starts without any of it. That is the argument for a Dockerfile, which is where the track goes next.
000 is a diagnosis
The AWS task: xfusion-ec2 runs nginx, the security group allows 80, the instance is running with a public IP, nothing reaches it.
The first command was not an AWS call.
curl -s -m 8 -o /dev/null -w "http_code=%{http_code}\n" http://98.82.179.65/
http_code=000
000 from curl means no HTTP response at all: the TCP connection never established. That single number rules out an entire category of causes before touching the AWS API.
A 403 or a 502 would mean packets arrive and something answers, which makes it an nginx or application problem. 000 means nothing is listening, or nothing can reach what is listening. Network path. Eight seconds well spent.
Sweep everything, then change one thing
An inbound packet to an EC2 instance crosses five checkpoints, and any one of them breaks it identically from outside: an internet gateway attached to the VPC, a default route pointing at it, the subnet associated with that route table, a NACL allowing the port, and a security group allowing the port. Plus a public IP, plus a process actually listening.
I checked all of them before changing anything. That discipline is the whole lesson, and it is not about being thorough for its own sake. Guess-and-check on a VPC means making changes to things that were never broken, and by the time it works you cannot say what fixed it or whether you left something worse behind.
The sweep found this:
{ "Dest": "0.0.0.0/0", "GW": "igw-0cf9ecd79f552b081", "State": "blackhole" }
blackhole is the most informative word in the entire output. It means the route exists and its target does not resolve: the gateway is deleted, detached, or otherwise unusable. AWS is telling you precisely what is wrong, in a field most people scroll past because the route itself looks correct.
Everything else was clean. The security group allowed 80 from anywhere. The NACL was the default open configuration. The instance was running with a public IP. The subnet was associated with the right route table. One thing was broken, and it announced itself.
Detached and deleted look identical
aws ec2 describe-internet-gateways --internet-gateway-ids igw-0cf9ecd79f552b081
# { "Attachments": [] }
The gateway existed with no attachments. Detached, not deleted. So the fix was one command:
aws ec2 attach-internet-gateway --internet-gateway-id igw-0cf9ecd79f552b081 --vpc-id $VPC
And the route went from blackhole to active on its own. No route edit, no re-association. The route was always valid; its target simply became reachable again.
Had the gateway been deleted, this would have been three commands, because a new IGW gets a new ID and the existing route would still point at the dead one. Same symptom, same first diagnosis, different remedy, and one describe call to tell them apart.
Why creating the missing thing would have failed
This is the part worth the post. The obvious move on "make it accessible" is to create an internet gateway and attach it, since the VPC visibly had none. That would have succeeded as a sequence of API calls and left the site broken.
The existing route points at igw-0cf9ecd79f552b081 specifically. A different gateway does not change what that route targets, so it would have stayed blackhole. You would end up with a working IGW, an orphaned one, a still-broken site and no idea why. A VPC also permits only one attached internet gateway, so the second attach fails once the first succeeds.
Creating the missing resource is not the same as fixing the broken reference.
There was a red herring too, worth naming because it is exactly the kind of thing that pulls you off course:
MapPublicIpOnLaunch: False
That looks wrong for a public subnet. It is not the problem. It only controls whether instances launched into the subnet later get an automatic public IP, and this instance already had one. Changing it would have been a no-op, and fixing incidental things during an incident makes it harder to say afterwards what actually resolved it.
Then close with the command you opened with:
curl -s -m 10 http://98.82.179.65/
# <title>Welcome to nginx!</title>
Same command, different answer. That symmetry is the point: the closing check should be the opening check, so the fix is demonstrated rather than assumed.
Everything told me
System has not been booted with systemd names its own cause. http_code=000 says the connection never opened. blackhole says the route's target does not exist. Three different systems, three precise messages, all of them easy to skim past in favour of a guess.
There is one thing I still want to know that the task did not ask for. An internet gateway does not detach itself, so somebody or something detached it, and that is a CloudTrail question. Fixing the symptom without finding out means it happens again.
So here is the Day 40 question. The last outage you resolved, can you name what caused it, or only what you changed?
Day 40 down. Sixty to go.
Top comments (9)
000covers four different diagnoses, and two of them point away from the network path rather than toward it. Same-winvocation on curl 8.7.1 against four local arms: nothing listening givesexitcode=7withtime_connect=0.000000, an unresolvable host gives6and also0.000000, a socket that accepts and then hangs gives28withtime_connect=0.000413, and one that accepts and resets gives56with0.000572. All four printhttp_code=000.The last two are a TCP connection that established and a server that then misbehaved, which is the nginx-or-application category the number is supposed to rule out. So the eight seconds buy less than the post credits them with, and an nginx wedged on a worker would have sent you through the same five-checkpoint VPC sweep.
The upgrade is free because it is the command you already ran: add
exitcode=%{exitcode} time_connect=%{time_connect}to the-wstring, and a non-zerotime_connecttells you the packets arrived before you open the AWS API at all.You're right, and this goes deeper than a wording fix.
I wrote that 000 means the TCP connection was never established. It doesn't. It means no HTTP status line came back, and exits 28 and 56 both produce it with a connection that established perfectly well. Which is exactly the nginx-or-application category I claimed the number ruled out.
So the post contains its own failure mode. An article about signals answering a narrower question than the one you asked, built on a signal answering a narrower question than I asked. I'd be more annoyed if it weren't so neat.
An nginx wedged on a worker accepts the connection, then hangs, prints 000, and I'd have walked the whole five-checkpoint VPC sweep hunting for a break in a path that was working fine. Eight seconds spent ruling out a category that hadn't been ruled out.
Adding exitcode and time_connect to the -w string, as you said. One note for anyone copying it: %{exitcode} arrived in curl 7.75.0, so on anything older it's $? after the call instead.
Correcting the post today. Thank you for testing it rather than asserting it.👊
Nnamdi, 40 days of daily DevOps posts — that is the kind of discipline I'm trying to build in my own practice. 🙏
Your line "Curl's 000 says where to look, Blackhole says what broke" hit me directly. I'm 12, building an AI coding mentor on a phone, and last week my Groq API model got deprecated silently. My fallback chain caught it, but it was my Supabase error_logs table (with latency_ms and status codes) that told me exactly what broke — my version of your "Blackhole."
The "where to look" part is what I'm still learning. When you have 40 days of debugging war stories, what's the single most underrated signal you check first when a system fails silently?
Building KODA from Tamil Nadu on a POCO C55. Your DevOps series is required reading. 🐯
Good question.
The most underrated signal is the count of things that succeeded, not the log of things that failed.
Error logs are reactive by definition. They catch what raised. Silent failure is the category where nothing raises, so an error table structurally cannot see it, however good the table is. Your error_logs with latency_ms is solid instrumentation, and it still has that hole in it.
Your Groq deprecation is the perfect example. The fallback chain caught it, so the system kept working and nothing shows any error. Which means the only thing that could have told you the primary had died is a record of which model actually served each request. Log that field, and a silent deprecation stops being invisible. It becomes a distribution change. Primary served 95% of calls last week and 0% today, with no errors in either week.
So two things I'd add to what you already have.
Log successes with a timestamp, not just failures. Then "when did this last work" is one query instead of an investigation, and the window between last success and first suspicion tells you where to look.
And log which path served the request. Fallback chains, retries, cache hits. A system that silently degrades is still succeeding, just differently, and recording how it succeeded is the only way to see that.
The general form: error monitoring answers what broke. Silent failure needs what changed, and that question needs the successful case recorded too.
Building this on a POCO C55 from Tamil Nadu is more impressive than the 40 days, for what it's worth. Keep going. 🙌
The curl 000 point is a useful clue, not an error category. I usually split the check into DNS resolution, TCP connect, TLS negotiation, and the final HTTP status, then run the same request from inside and outside the container. That makes a proxy, security-group, or missing route much easier to isolate.
Splitting by phase is the right frame, and running it from inside and outside the container is the part I don't do often enough. That's what separates a proxy or an egress rule from a genuine path problem, and I've misdiagnosed the first as the second before now.
Between your four phases and @vinhnguyenthanhdn's exit codes, the write-out string does most of it in one call. time_namelookup, time_connect, time_appconnect, exitcode, http_code. One request, five answers, and you know which phase broke before you open a console.
Wow 40 days complete ✅
@technogamerz 40 down, 60 to go. 🎉 Thank you!
Though today's post is mostly me being corrected in the comments, so that 40 is doing a lot of heavy lifting. 😄
Some comments may only be visible to logged-in visitors. Sign in to view all comments.