My most popular page does not exist

What this is about. While working out why my new robot counter had such a
flattering opinion of my readership, I read the raw server log and found that
the most requested page on the whole machine was one that has never existed.
This is what I did about it, and the mistake that nearly took all my sites
offline.

Why. Building the trap took an afternoon. Working out what must never be
trapped took longer, and that is the part worth writing down.

Fifteen thousand requests for nothing

/wordpress was the most requested path on the server for my newly deployed
site. More than the homepage. Fifteen thousand requests in a single log file.

I have never run WordPress there (!). I once had a React front end that read
content from WordPress on a different machine entirely, which is not the same
thing and would not leave this trail.

Every one of those requests forged a browser user-agent, rotating through
Chrome, Edge, Safari, Firefox and Opera in almost exactly equal numbers. That is
one scanner with a costume box, not many visitors. It is also the kind of sneaky
little robot my shiny new counter cannot see, because it is pretending to be a
person. Once identified, never doubted. No amount of counting fixes that.

What it can see is that the requests came from a couple of dozen addresses, two
of which accounted for 87 percent of them.

Twenty bytes a second

A tarpit is an old idea (a pit and a load of tar): answer, but answer so
irritatingly slow that the other side wastes real time. Ha! So I gave them
something to download. A file called wordpress_backup.sql, served at twenty
bytes per second. The whole megabyte takes just over fourteen hours.

The sketch I started from bolted on a Python service whose entire job was to
dribble ones and zeros forever. The web server does it alone, sulkily, for
nothing: a finite file served slowly enough is indistinguishable from an
infinite one, so there is no extra process, no extra port and nothing extra to
keep patched. I also dropped a (for me) clever header containing a JavaScript
infinite loop, on the grounds that response headers are not executed by
anything.

Then the web server and I fell out over what "slow" means.

The rate limit throttles the headers too. A client spends eighteen seconds
receiving about 360 bytes of response headers, so it is stuck before it even
learns the filename it is being offered. I spent a while convinced my header
configuration was broken, when in fact it was still in flight. Stained by tar up
to my knees already...

The defaults make a worse trap than small buffers do. Out of the box the
server hands over one four-kilobyte chunk immediately and then goes quiet for
about three and a half minutes, which any scanner with an inactivity timeout
treats as a dead connection and abandons. Shrinking the buffers turns that into
a steady dribble, so the connection stays interesting enough to sit in. Smaller
buffers, stickier trap.

Up to my elbows in tar

Then I widened it to the other things robots ask for. /.env, /.git/config,
the files that hand over credentials when they exist. All of those begin with a
dot, so a rule for dot paths looked obvious.

So does /.well-known/acme-challenge/, which is how my TLS certificates prove
they are mine when they renew.

A rule that trapped every dot path would have tarpitted my own certificate
renewals. Nothing would have broken that day, or that week. It would have
surfaced a month or two later as expired certificates on every site at once, and
I would never have connected it to the "clever" thing I did in October.

There is a second exclusion of the same shape. Search engines speculatively ask
every site for /wp-json/ and /wp-sitemap.xml, just in case WordPress is
hiding there. A blanket rule for anything starting with wp would have put
Googlebot in the tarpit for fourteen hours at a time. I first assumed those
requests were leftovers from some old setup and would fade. They are not. They
are standard crawler behaviour and they will keep arriving forever (or until
Mullenweg goes crazy again), so the exclusion is permanent. The note in the
config now says so. The version where it looks temporary is the version somebody
deletes at the Spring cleanout.

What it has caught

So far the trap has held 91 requests for a combined 25.6 hours. The average
stay is seventeen minutes. Twenty lasted more than ten. The longest single
connection sat there for eight hours and ten minutes.

The funniest part is that it is barely catching WordPress scanners at all. The
top catches are /.env and /.git/config. I built a little black sticky goo
for one kind of robot and it filled up with another. Never thought slow
connections could bring me so much joy :)

What happened, and what else I could have done

A page that does not exist was my most popular page. It now answers, very
slowly, to everyone except the three kinds of visitor that must never be slowed
down: real people, search engines doing their normal speculative sniffing, and
my own certificates renewing themselves.

A tarpit is not the only answer, and it is not obviously the best one:

The honest caveat is that a tarpit costs me something too. I hold a connection
so that they hold one, and if the scanning came from thousands of addresses
instead of a couple of dozen, that trade would stop favouring me. Hence a hard
ceiling on how many of these connections can exist at once, and a refused one
being closed rather than answered.

And a small confession about measurement, which is becoming a theme. My first
instinct was to keep these requests out of the logs entirely, since fifteen
thousand lines a day is noise. I did exactly that, and then realised I had built
a trap with no way of telling whether it caught anything. They now go to their
own log file, which is how I know about the eight hours.

Hit me in the comments if you think my reasoning is flawed, or if you think bots
should be treated as equals.

Originally published at idunworks.com.