I failed today's AWS task on the first attempt, with every resource created correctly and every status field green. That is the more useful half of this post, so it gets the space.
The Docker half sets it up neatly, because both are about the gap between a label and the thing it labels.
One Docker task, one AWS task. Pull an image and give it a second tag, then build and run a container on ECS Fargate. The tasks come from the KodeKloud Engineer platform.
docker tag copies nothing
docker pull <image>:<tag>
docker tag <image>:<tag> <image>:<new-tag>
docker images
Run that and docker images shows two rows, two different tags, and one identical IMAGE ID. Disk usage does not move. docker tag did not duplicate anything; it added a second name pointing at the same image.
Once that lands, two other behaviours stop being surprising. docker rmi on one of two tags prints Untagged: rather than Deleted:, because removing a name is not removing an image, and the layers survive while anything else references them. And an image reference is really [registry/][namespace/]repository[:tag], so docker pull ubuntu silently means docker.io/library/ubuntu:latest.
That last one is the bridge to the AWS half, and it is worth stating plainly: the registry is part of the name. Not a setting, not a flag, part of the string.
The other thing worth carrying: a tag is a mutable pointer. The same tag can point at a different image next month, and docker pull on an unchanged tag can return different layers. image@sha256:... pins a digest and is the only reference that is genuinely immutable.
Every resource correct, task failed
Six resources: an ECR repository, an image built and pushed, a Fargate cluster, a task definition, a service, and a running task.
All six came up clean. The service read ACTIVE. runningCount matched desiredCount. The task read RUNNING with no stoppedReason. The container had pulled from ECR without complaint. I checked all of that, concluded it worked, and submitted.
Application is not accessible
Nothing had opened port 80. The service used the default security group, which permits inbound traffic only from itself. assignPublicIp=ENABLED had given the task a public IP, so it could reach out to ECR to pull the image, but security groups are stateful and directional: outbound worked, inbound never did. nginx was listening on a routable address that no packet could arrive at.
One rule, and it belongs before the service is created rather than as a repair afterwards:
aws ec2 authorize-security-group-ingress \
--group-id $DEF_SG --protocol tcp --port 80 --cidr 0.0.0.0/0
This is the same rule I wrote about two days ago as the load-bearing part of the ALB task. Recognising it in one task and not applying it in the next is the actual failure here. Nothing about it is ECS-specific.
What RUNNING actually claims
RUNNING is a statement about a container's lifecycle. The image pulled, the process started, it has not exited. It says nothing whatsoever about whether anything can reach it. Two independent facts, and only one of them was what the task asked for.
The rule I took from it: the final check has to exercise the thing the task promises, not the thing that is easiest to query. A status field is easy to query. It is not the deliverable.
For Fargate that check is slightly awkward, which is probably why it gets skipped. A task does not carry its IP; it gets its own network interface, and the address is two lookups away:
ENI=$(aws ecs describe-tasks --cluster datacenter-cluster --tasks $TASK_ARN \
--query "tasks[0].attachments[0].details[?name=='networkInterfaceId'].value" --output text)
TASK_IP=$(aws ec2 describe-network-interfaces --network-interface-ids $ENI \
--query 'NetworkInterfaces[0].Association.PublicIp' --output text)
curl -s -m 10 http://$TASK_IP/
With EC2 the public IP is one field on the instance. Here it is task to ENI, ENI to address. Worth building the habit anyway, because without a load balancer in front it is the only way to prove the deployment works.
Three ECS specifics that cost real time
The registry is the address. docker push datacenter-ecr:latest is not a misconfigured push to ECR, it is a push to Docker Hub under a repository of that name, and it fails on authorization moments after you watched ECR say Login Succeeded. Re-tag with the full URI first. Same fact as the Docker half, in a place where it bites.
Two roles, and they are not the same job. executionRoleArn is used by the Fargate infrastructure before your container exists: it pulls the image and creates log streams. taskRoleArn is used by your application code once it is running. The trust principal for both is ecs-tasks.amazonaws.com, not ecs.amazonaws.com. Both principals exist and mean different things, and naming the wrong one produces a role that looks correct in the console and can never be assumed by a task.
CannotPullContainerError has two unrelated causes. A missing or wrong execution role produces it. So does a task with no route to ECR, because Fargate pulls the image over the network like any other client. The distinguishing detail is the wording underneath: an authorization failure means the role, a timeout means networking. Reading only the headline sends you to the wrong half of the problem.
And one piece of pure trivia that is not trivia when it fails. awsvpcConfiguration={subnets=[...]} takes a comma-separated list. --subnets on create-load-balancer takes space-separated values. Two conventions in the same CLI, and mixing them gives you a parse error rather than anything that explains itself. I found that one by getting it wrong rather than by reading it anywhere.
The label is not the thing
A tag names an image, and adding one copies nothing. RUNNING names a lifecycle state, and it promises nothing about reachability. In both cases the label is accurate and answers a narrower question than the one being asked of it.
The cost of the first is a moment of confusion in docker images. The cost of the second was a failed submission on work that was otherwise complete.
So here is the Day 38 question. When you call something done, is it because the deliverable responded, or because a field said the word you were hoping to see?
Day 38 down. Sixty-two to go.
Top comments (5)
"Every resource created correctly and every status field green" is the defining failure mode of cloud work: the console's green means "the resource exists in the desired state", and existence is not reachability. RUNNING is a label about the task; reachable is a fact about the path - security group, listener rule, health check grace period, the target group actually having registered targets. The check that would have caught it isn't another status read, it's a request through the real path: curl the service from outside, because that's the only vantage point your users have. Same lesson as the tag: a tag is a name you gave a digest, not a copy of anything - labels are free and therefore lie for free. Verify the thing, never the label.
"Labels are free and therefore lie for free" is going straight into my notes.
And I'd push the same discipline one level up, because a request makes a narrower claim than it looks like it does.
A successful curl proves one path, from one place, at one moment. Put a load balancer in front with three targets, and your request hits one of them. If two are broken, you stand a decent chance of a green result that's wrong, and you'll have verified the deliverable in exactly the way the post recommends. Stale DNS pointing at the previous deployment does the same thing. So does a cached response. So does hitting a health check page rather than the application, which is its own trap when / returns something and the app lives elsewhere.
None of that makes your rule wrong. It makes it recursive. A status field answers a narrower question than the one you asked, and so does a single request, just less narrowly.
Where I've landed is that the check has to be the thing that fails when the thing is broken, and that's usually more than one request. Repeat it, hit each target directly, and know which path you actually exercised.
Though I'll take a request over a status field every time. Better class of lie.
Nice write-up Nnamdi!
Thank you! Divya 😄 This one was humbling to write. Everything was green, everything was created correctly, and I still failed it because I checked the status field instead of the thing the task actually asked for.
Turns out the more useful posts are the ones where I got it wrong.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.