Day 55: The Volume Hides nginx's Log Links, and Every Azure Subnet Gives Up Five Addresses

Day 55 of DevOps, Day 5 of Azure. In both tasks today, the interesting part was something that was already there before I wrote a line.

The nginx image had already decided where its logs go, and the sidecar task only works because a volume quietly overrides that decision. And every Azure subnet has five addresses taken before anything is deployed into it.

One Kubernetes task, one Azure task. Run a sidecar container that reads nginx's logs, then create a small virtual network with a fixed address range. The tasks come from the KodeKloud Engineer platform.

A sidecar that reads nginx's logs

volumes:
- name: shared-logs
  emptyDir: {}
containers:
- name: nginx-container
  image: nginx:latest
  volumeMounts:
  - name: shared-logs
    mountPath: /var/log/nginx
- name: sidecar-container
  image: ubuntu:latest
  command: ["sh", "-c", "while true; do cat /var/log/nginx/access.log /var/log/nginx/error.log; sleep 30; done"]
  volumeMounts:
  - name: shared-logs
    mountPath: /var/log/nginx

Two containers share nginx's log directory through an emptyDir, and the sidecar prints both log files every 30 seconds. Make a request, and it shows up in the sidecar's output:

kubectl exec webserver -c nginx-container -- curl localhost
kubectl logs webserver -c sidecar-container

The task is short. The reason it works is the interesting part.

Why it works at all

The official nginx image does not keep log files. Its Dockerfile links them to the container's output streams:

# forward request and error logs to docker log collector
    && ln -sf /dev/stdout /var/log/nginx/access.log \
    && ln -sf /dev/stderr /var/log/nginx/error.log \

nginx's packaged configuration writes to exactly those two paths, so in a plain nginx container, every log line goes straight to stdout or stderr. There is nothing on disk for a sidecar to read.

Mounting the volume at /var/log/nginx changes that. An emptyDir starts empty, and a mount hides whatever the image had at that path. The Kubernetes docs, describing a ConfigMap volume, say the mount makes files the image had in that directory inaccessible, and Docker's docs describe pre-existing files as "obscured by the mount". With the links hidden, nginx creates ordinary files on the shared volume, and those are what the sidecar reads.

So the volume does two jobs. It gives the sidecar something to read, and it takes nginx's logs away from nginx's own output. kubectl logs shows what a container writes to stdout and stderr, so after this change, the nginx container's own log should show no requests at all. They only come back through the sidecar.

Related trap: kubectl logs webserver with no -c shows only one container. kubectl uses the pod's default-container annotation if there is one, and otherwise the first container, which here is nginx, not the sidecar. In a pod with more than one container, always pass -c.

The pattern, and its cost

Kubernetes' logging docs describe this streaming sidecar pattern, and they are candid about the price: "writing logs to a file and then streaming them to stdout can double how much storage you need on the node." For an application that writes to a single file, the same docs recommend setting /dev/stdout as the destination instead, which is exactly what the nginx image already did.

So this lab builds the pattern around an application that did not need it. It is still worth learning, because plenty of software can only write to files, and a sidecar is how you bring that output back to where the cluster collects logs. My version also re-prints both files in full every 30 seconds, so every line repeats on every pass. A real shipper follows the file instead.

Kubernetes also has native sidecars now, stable since v1.33: an init container with restartPolicy: Always, which starts before the app container, keeps running, and is stopped after it. The two regular containers I used are still valid, and the docs describe that form as appropriate when you do not need to control which container starts or stops first.

Five addresses you never get

The Azure task was a VNet, nautilus-vnet, with a fixed range of 192.168.0.0/24. Yesterday's was 10.0.0.0/16. That is 256 addresses against 65,536, and the smaller number shrinks further once you carve it into subnets.

Azure reserves five addresses in every subnet:

A /24 subnet leaves 251 usable addresses. The smallest subnet Azure supports is a /29, which leaves 3.

AWS reserves the same five positions, with slightly different jobs: .1 is the VPC router, .2 the DNS server, .3 is held for future use, and the last address is reserved because AWS does not support broadcast. The arithmetic carries over from the AWS track. The size limits do not, since AWS subnets stop at /28.

What a /24 cannot hold

Some Azure services need their own subnet, with a fixed name and a minimum size:

Two /26 subnets take half of a /24 before a single VM exists. A VNet that might one day need a firewall and Bastion should not start this small. My notes had the gateway subnet as "/29, /27 recommended"; Microsoft's current page makes /27 the minimum for everything except the Basic SKU.

One more correction to my notes. I called the three RFC 1918 blocks the ranges Azure accepts. They are the ranges Microsoft recommends. Azure also treats the RFC 6598 range, 100.64.0.0/10, as private, and it allows public ranges too. Those are a bad idea for a practical reason: Azure routes every range in a VNet's address space inside the VNet, so the real internet hosts at those addresses become unreachable from it.

What was there before you

The nginx image made a choice about logs before I wrote any YAML, and my volume overrode it without saying so. Azure took five addresses out of every subnet before I deployed anything. Neither is hidden. Both are in the documentation, and both change what your own configuration actually does.

So here is the Day 55 question. What does the base image you use most already do that your configuration quietly depends on, or quietly undoes?

Day 55 down. Forty-five to go.