I posted a broken load-balancer config to Caddy's admin API and it told me the reload had worked. The server behind it stopped answering requests entirely. The API still said 200.
That was on Caddy 2.11.4. I wanted to know whether the two patch releases that followed it, 2.11.6 (1 October) and 2.11.7 (3 October), actually closed that gap, because their changelogs claim "stricter config validation" and a fix for a Unix socket that used to hang on reload. Both claims turned out to be true, and more precise than I expected on the first one and more dramatic than I expected on the second.
The config that lied about succeeding
Caddy's admin API has a /load endpoint: POST a JSON config, it replaces the running one. I sent this, which gives a weighted_round_robin policy a negative weight:
"load_balancing": {
"selection_policy": {
"policy": "weighted_round_robin",
"weights": [1, -3, 2]
}
}
On Caddy 2.11.4:
$ curl -s -o /tmp/out.json -w 'HTTP %{http_code}\n' -X POST \
-H 'Content-Type: application/json' --data @bad_weighted.json \
http://localhost:2019/load
HTTP 200
Then I curled the server's actual listener on port 8080:
$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://localhost:8080/
curl: (52) Empty reply from server
The admin API said the new config was live. The thing it was supposedly serving had stopped responding. On Caddy 2.11.6 and 2.11.7, the same POST gets this instead:
HTTP 400
{"error":"loading config: loading new config: loading http app module: provision http:
server srv0: ... http.reverse_proxy.selection_policies.weighted_round_robin:
invalid configuration: weighted_round_robin requires at least one upstream with a
positive weight"}
And the server that was already running kept running. I reloaded a known-good config on 2.11.7, confirmed it was serving, sent the broken one, got the 400, and checked again:
| Step | 2.11.4 | 2.11.7 |
|---|---|---|
POST bad config to /load
|
200 |
400 with error body |
| Live traffic afterwards | empty reply, server down | unaffected, still serving old config |
If you have anything automated that pushes Caddy config over this endpoint and only checks for a 2xx, on 2.11.4 and earlier that check was lying to you.
Caddy logs every admin API request, which is the other way to catch this if you're watching logs rather than status codes. On 2.11.4, the log line for the bad load reads:
{"logger":"admin.api","msg":"load complete"}
On 2.11.7, the same POST produces:
{"level":"error","logger":"admin.api","msg":"request error","error":"... weighted_round_robin:
invalid configuration: weighted_round_robin requires at least one upstream with a positive
weight","status_code":400}
"Load complete" is exactly what it sounds like: as far as 2.11.4 is concerned, nothing went wrong. A log-based alert watching for admin.api errors would have caught this on the newer version and found nothing at all to catch on the older one, same as checking the status code would.
It's a specific list, not a general re-check
The changelog names several newly-rejected categories, and I tested two more of them directly, because I wanted to know whether "stricter" meant "all of it" or "this list, no more."
A Caddyfile with a duplicate uri inside forward_auth:
forward_auth 127.0.0.1:9099 {
uri /auth1
uri /auth2
}
On 2.11.4, caddy adapt accepts this with no warning and silently keeps the second value. On 2.11.7:
$ caddy adapt --config Caddyfile.forwardauth
{"level":"error","msg":"parsing caddyfile tokens for 'forward_auth': cannot re-declare uri: /auth2, at /Caddyfile.forwardauth:4"}
Exit code 1. A map handler with a destination placeholder missing its closing brace behaved the same way: 200 on 2.11.4, and on 2.11.7:
{"error":"... http.handlers.map: destination 0 must be a placeholder and only a
placeholder, but got '{http.vars.unterminated'"}
What I got wrong on the way
I assumed "stricter validation" meant any previously-accepted mistake would now be caught, so I tried two more cases expecting the same result and got surprised twice.
First, duplicate @id values on two routes. I expected this to be a new catch. It wasn't — Caddy 2.11.4 already rejects it:
{"error":"loading config: indexing config: duplicate ID 'r1' found at
/config/apps/http/servers/srv0/routes/0 and /config/apps/http/servers/srv0/routes/1"}
Same error, same HTTP 400, on both versions. Second, a file_limit set to 3.5 where the schema wants an integer. Also already rejected on 2.11.4, by Go's own JSON decoder rather than anything Caddy added:
{"error":"... json: cannot unmarshal number 3.5 into Go struct field
Browse.browse.file_limit of type int"}
So the fix is a named list of specific gaps — duplicate named_routes, duplicate forward_auth subdirectives, bad weighted_round_robin weights, ambiguous map inputs, and a handful of others — not a blanket pass over every handler. Worth knowing before you assume a config mistake you haven't tested is now covered.
I also checked a boundary case on the weight fix, because "stricter" sometimes means "stricter than it should be." A weight of 0, as opposed to negative, is accepted on 2.11.7:
"weights": [1, 0, 2]
returns 200. The rule is "at least one positive weight," not "no zeroes," so taking an upstream fully out of rotation with a zero weight still works.
The Unix socket that took two minutes to let go
This is the one I didn't expect to be able to measure cleanly, and it was the most striking result of the day.
The 2.11.7 changelog says a reload that moves a server's listener off a Unix socket used to leave the old socket file in place and leave clients connecting to it hanging for "about two minutes," fixed by closing it immediately. I set up Caddy 2.11.6 listening on unix//tmp/sock_a, confirmed it answered, then POSTed a new config that moves the same server to unix//tmp/sock_b:
$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' -X POST --data @b2.json \
http://localhost:2019/load
HTTP 200
Then I polled the old socket path every half second or so. For roughly the first 100 seconds, every attempt timed out rather than failing fast:
curl: (28) Operation timed out after 1001 milliseconds with 0 bytes received
Around the 110-second mark it flipped to a connection reset, and by about 120 seconds it was a clean, instant refusal. The socket file itself (/tmp/sock_a) was still sitting on disk the entire time — ls kept finding it right up until the connections started failing fast. That is close enough to the changelog's "about two minutes" that I don't think I got lucky with the timing; it's a real, repeatable window.
On Caddy 2.11.7, running the identical reload:
$ ls /tmp/sock_a
ls: cannot access '/tmp/sock_a': No such file or directory
$ curl -s --unix-socket /tmp/sock_a http://localhost/
curl: (7) Failed to connect to localhost port 80 after 0 ms: Couldn't connect to server
The file was gone and the refusal was instant, checked immediately after the /load call returned and ten more times over the next three seconds. No timeout anywhere in that window.
| 2.11.6 | 2.11.7 | |
|---|---|---|
| Old socket file after reload | still present | removed immediately |
| Connecting to old socket | hangs ~100-120s, then reset/refused | refused instantly, 0ms |
If you reload Caddy in a loop as part of a deploy pipeline and something downstream reconnects on a fixed interval, the old version could leave that something stuck for two minutes against a socket nobody is serving on any more. That is a real production cost, not a cosmetic log message.
Run it yourself
This needs Docker and about ten minutes once the images are pulled.
docker pull caddy:2.11.4 && docker pull caddy:2.11.7
cat > good.json <<'EOF'
{"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"routes":[
{"handle":[{"handler":"static_response","body":"hello-good"}]}
]}}}}}
EOF
# Deliberately invalid: a negative weight
cat > bad_weighted.json <<'EOF'
{"apps":{"http":{"servers":{"srv0":{"listen":[":8080"],"routes":[{"handle":[{
"handler":"reverse_proxy",
"load_balancing":{"selection_policy":{"policy":"weighted_round_robin","weights":[1,-3,2]}},
"upstreams":[{"dial":"127.0.0.1:9001"},{"dial":"127.0.0.1:9002"},{"dial":"127.0.0.1:9003"}]
}]}]}}}}}
EOF
for v in 2.11.4 2.11.7; do
docker run -d --name caddy-$v caddy:$v caddy run --resume=false >/dev/null
sleep 1
docker cp good.json caddy-$v:/good.json
docker cp bad_weighted.json caddy-$v:/bad_weighted.json
echo "== $v =="
docker exec caddy-$v curl -s -o /dev/null -w 'load good: HTTP %{http_code}\n' \
-X POST -H 'Content-Type: application/json' --data @/good.json http://localhost:2019/load
docker exec caddy-$v curl -s -o /dev/null -w 'traffic after good: HTTP %{http_code}\n' \
http://localhost:8080/
docker exec caddy-$v curl -s -o /dev/null -w 'load bad: HTTP %{http_code}\n' \
-X POST -H 'Content-Type: application/json' --data @/bad_weighted.json http://localhost:2019/load
docker exec caddy-$v curl -s -o /dev/null -w 'traffic after bad: HTTP %{http_code}\n' \
http://localhost:8080/
docker rm -f caddy-$v >/dev/null
echo
done
This is the actual output from running it:
== 2.11.4 ==
load good: HTTP 200
traffic after good: HTTP 200
load bad: HTTP 200
traffic after bad: HTTP 000
== 2.11.7 ==
load good: HTTP 200
traffic after good: HTTP 200
load bad: HTTP 400
traffic after bad: HTTP 200
On 2.11.4, the bad config reports 200 and the server stops answering (000, connection failure). On 2.11.7, the bad config is refused with 400 and the server carries on serving hello-good exactly as before. The /load response alone tells you which version you're running, without needing a second request against the data path at all.
What to do with this
If anything you run pushes configuration to Caddy's admin API — a CI job, a control plane, a sidecar — and it treats a 2xx from /load as "the new config is live and correct," go check which version you're running. Before 2.11.6, that check could pass while the server underneath it was already broken. After 2.11.6, it still can't tell you about every possible mistake, only the specific ones listed in the changelog, so don't drop your own post-reload health check just because the API got stricter.
And if you reload Caddy on a schedule and something else in your infrastructure dials a Unix socket Caddy used to own, check your Caddy version before assuming a two-minute stall is your code's fault.
Top comments (4)
The 200 that lies is the part of this I would steal for every control plane I touch, because a reload endpoint that reports success while the server underneath is dead is worse than a loud failure — nothing pages, nobody looks, and the first person to notice is a user. My rule from running config reloads is to never trust the control-plane response alone: after every reload I probe the data path and compare what is actually being served, not what the deploy log claims. The stale-socket hang is the same shape in a different costume — a two-minute hang on a socket nobody serves anymore looks exactly like a slow downstream, so the on-call chases the wrong service for two minutes. I learned that one in document pipelines, where a config change that "applied" but did not take effect only showed up in the output, never in the logs. Do you probe the old socket path after a reload too, or only the new listener?
Yeah "probe the data path, not the deploy log" is pretty much where the post ends up too, the 400 fixes a specific list of mistakes, not all of them.
On the old socket, in the test I did probe it, that's the only reason the two minute window showed up at all. I polled the old path every half second or so after moving the listener, and on 2.11.6 the file was still sitting there with every connect timing out until somewhere around 100 to 120 seconds. If I'd only checked the new listener everything would have looked fine, since the new socket was answering straight away.
So if a reload moves a listener, I'd check both. The new one should answer, and the old path should either be gone or refuse instantly. A short connect timeout on the old path is the useful bit, a refusal comes back in 0ms, a hang just eats your timeout, so you know which one you got right away.
The half-second poll is the detail that makes the whole thing checkable, and the refusal-versus-hang distinction is the part I'll steal. In my document pipelines the equivalent trap was a config change that 'applied' but only showed up in the output, never the logs, so I learned to probe the thing that should be dead, not just the thing that should be alive. Your 100-to-120-second linger is a good reminder that a dead socket that hangs is worse than one that refuses, because the hang eats the timeout budget silently. I'll keep the short-timeout probe on the old path as a standing check after any listener move.
yep, "probe the thing that should be dead" is a good way to put it. cheers