Telling censorship from outages: probing Russia's internet from both sides of the border

When something stops loading in Russia, there are two very different explanations. The service might be down for everyone. Or it might be up everywhere except inside Russia. Outage trackers built on user complaints can't tell these apart: a spike of "YouTube isn't working" looks the same in both cases.

I built shutdown.fyi, an open tracker for 38 services, 8 carriers and 85 regions. The core idea is simple: check the same address from inside Russia and from abroad, step by step, and compare where each attempt stops.

Two vantage points, four steps

Every 5 minutes, for every service, we probe up to two hostnames. We use the real websites, not CDN hosts: a TLS handshake to an image CDN tells you nothing. The steps are:

1. DNS — does the name resolve, and to what?
2. TCP — can we open a connection to port 443?
3. TLS — does the handshake with the correct SNI complete?
4. HTTP — do we get a response?

The same sequence runs from a server in Russia and from one in Helsinki. The verdict comes from the difference:

A real result for youtube.com today:

"ru":     { "ip": "142.251.155.4", "pingMs": 20, "tcp": "ok", "tcpMs": 20, "tls": "timeout", "http": null },
"abroad": { "tcp": "ok", "tls": "ok", "tlsMs": 7, "http": 200 }

TCP connects in 20 ms, so the route is fine and the server is up. Then the TLS handshake just hangs, but only from Russia. That pattern (blocked-tls) is what filtering on the server name in the TLS ClientHello looks like. Telegram, WhatsApp and Instagram fail one step earlier (blocked-tcp): the IP addresses themselves are unreachable.

Two axes instead of one status

This forced a design decision. A status page usually has one colored dot. Here that's a lie in either direction:

• a service restricted for months is "nothing new", so an anomaly detector shows it as normal;
• the same service is obviously not reachable, so a reachability check shows it as broken forever.

So every entity has two independent axes. Access is measured only by active probes and exists only for services; a carrier or a region has no "reachability" we can measure. Incident answers "is it worse than usual for this hour?" and comes from complaints, OONI measurements, IODA traffic data and BGP visibility.

"Usual for this hour" matters. Complaints at 3 a.m. and 8 p.m. differ by an order of magnitude. The baseline is the median of the same hour over previous days. It has a floor of half the weekly median, so three complaints on a quiet night don't read as "10× above normal".

Bugs that taught us something

Levels are not counts. We aggregate eight public outage detectors. Some publish an hourly series. Others publish only "complaints in the last 24 hours". We wrote that 24-hour number into the current hour on every run, so a daily sum added up 24 copies of a daily level. One source reported 18 messages a day on its own page, and we showed ~450. The fix: a rolling 24-hour level becomes 1/24 per hour. A day total plus a chart becomes "take the sum from the total, the shape from the chart".

The bank that wasn't blocked. Our Russian probe flagged a major Russian bank as "restricted": TLS timed out from Russia and worked from Helsinki. The bank isn't blocked in Russia. Its anti-bot protection drops connections from hosting IP ranges, and our probe lives in a datacenter. Domestic services on the government "whitelist" now get "no conclusion" instead of "restricted" when only our Russian vantage fails.

Flapping. Incidents closed on the first "ok" snapshot and reopened on the next spike. The feed filled with six-minute "outages". Now an incident closes only after 15 minutes of normal readings, and anything shorter is hidden.

ркн inside a word. The news matcher for "РКН" (the Russian telecom regulator) matched inside «подчеркнул» ("emphasized"), so a mayor's speech ended up on a status page. Word boundaries in Cyrillic need explicit lookarounds; \b doesn't know Cyrillic.

Open data

Everything is available as JSON without a key, CC BY 4.0:

curl -s https://shutdown.fyi/api/status/service/youtube.json | jq '.probe.hosts[] | {host, verdict}'

There's an embeddable widget too. Docs, examples and the full methodology are on GitHub: Chumbayoumba/shutdown-fyi-open.

If you do network measurement, I'd love feedback on the vantage-point approach. A single Russian vantage in a datacenter is the obvious weak spot, and residential probes would make the verdicts much stronger.