thread 'zql-connection' has overflowed its stack
exit code 3221225725 (STATUS_STACK_OVERFLOW)
The query that triggered that was SELECT 1+1+1+1+...+1, roughly one thousand seven hundred and fifty terms long, contained in under four kilobytes of text. There was nothing unusual about it: no malformed bytes, no adversarial encoding, just repeated addition. Yet it did not merely fail on its own terms. It brought down the entire process, so every other connection that happened to be open at the time died along with it.
I had written an entire sweep of tests whose purpose was to assert that nothing in this codebase ever panics: every canonical query truncated at every byte and mangled at every position, sixty-two deliberately corrupted database files, each case wrapped in catch_unwind as a last line of defense. None of it caught this failure, because a stack overflow is not a panic. It aborts the process outright, and catch_unwind never gets the chance to run.
That gap, between a test suite that asserts nothing panics and a server that is actually safe to run, is really what this project ended up being about, more than the SQL itself.
The premise
zql opens a .db file, a CSV file, or an entire directory, and lets you query all of it with SQL over the real PostgreSQL wire protocol. psql can connect to it without ever realizing it isn't talking to an actual Postgres server. Its Cargo.toml has an empty [dependencies] section, and cargo tree prints exactly one node.
$ zql ~/projects
zql 0.1.0, listening on 127.0.0.1:5432
$ psql -h 127.0.0.1
you=> SELECT ext, COUNT(*) FROM files GROUP BY ext;
psql itself represents roughly thirty years of someone else's C code. Watching it complete a full handshake and execute a real query against a Rust binary that imports nothing at all is, in a single screenshot, the entire pitch of this project.
What got reimplemented
Seventeen substitutions went into this project in total, against a stated bar of ten. Most of them were straightforward. Two, however, were genuinely difficult, because there was no shortcut available. Both the file format and the network protocol had to be understood directly from their specifications rather than from a library's documentation.
The SQLite reader parses the real file format on disk: pages, cell pointer arrays, variable-length integers, eleven distinct serial types, and the reassembly of overflow page chains, all without linking against libsqlite3. The first version of this reader got three separate details wrong, and all three were discovered by comparing its output against Python's own sqlite3 module rather than by rereading the specification more carefully. An INTEGER PRIMARY KEY column, for instance, is stored as NULL inside the record itself, because its real value lives in the cell header instead. Without handling that case specially, every primary key in every table came back as NULL, which is plausible enough to survive a casual test and wrong in every single row. The overflow threshold has a similar trap. SQLite does not wait until a page is completely full before spilling data into an overflow chain, because it deliberately keeps the tree less dense than that, so a reader that fills the page first before spilling will read small values correctly and larger ones silently wrong.
The wire protocol hides its own trap, and it is almost funny once you see it. The correct reply to a client's SSLRequest is a single unframed byte, not a proper message, even though every other exchange in this protocol follows the same pattern of a tag, a length, and a body. psql sends an SSLRequest before anything else, on every connection, without exception. Reply to it with a normally framed message and nothing throws an error anywhere in the stack. The connection simply sits open, waiting for a reply that will never come.
What the standard library made painful
There is no async runtime in the standard library, so every connection here runs on its own OS thread, which is a reasonable trade at this scale. There is no random number generator either, so the backend's cancellation secret is built from SystemTime nanoseconds mixed with a counter. It is guessable, and the README states that plainly rather than pretending otherwise. There is no time zone database, so every timestamp zql reports is in UTC, without exception. And there is no way to enumerate network interfaces, so telling a phone which address to connect to means opening a UDP socket, connecting it to a routable address without ever sending a packet, and reading back whichever interface the kernel actually chose for that connection.
None of these gaps is difficult in isolation. What is uncomfortable is how often each one turns into a limitation the user has to be told about, rather than an implementation detail that a library would otherwise have quietly absorbed.
The package that stopped looking necessary
Nearly every SQLite consumer I could think of links against the real libsqlite3 in C, or wraps a crate that does exactly that. zql does neither. There is no unsafe code anywhere in its source, and no extern block either.
grep -rn "unsafe" src/
grep -rn "extern" src/
Both commands return nothing. The obvious way to parse a binary file format quickly is to transmute a byte slice directly into a struct and trust that the bytes are what you expect. zql instead reads through bounds-checked slices the entire way through, which is the slower and more careful approach, and it is exactly why sixty-two corrupted database files produce clear error messages instead of crashes. Once that reader was working correctly, rusqlite, sqlx, and libsqlite3-sys all stopped being things the project actually needed. Not because those crates are poorly made, but because the underlying job, reading bytes according to a published specification, never actually required trusting a C library or an unsafe block in the first place.
The edge case that ate an afternoon, and then some
Back to the stack overflow.
Every phase that runs after parsing walks a SQL expression by recursion: binding, the GROUP BY fingerprint, evaluation, and even the drop code for Box<Expr> itself. Because tree depth
translates directly into stack depth, the fix has to live where the tree is built, in the parser, so that it closes off every later phase at once. That part of the fix was not the hard part.
Choosing the actual number was.
A limit of five hundred looked generous when tested against a release build, which in practice holds up to roughly one thousand seven hundred and fifty levels of depth. Then cargo test overflowed inside the parser itself, in a debug build, at a small fraction of that number. Debug frames are several times larger than release frames, and the gap is not even consistent across different expression shapes:
shape release debug
lower(lower(...)) ~1750 ~110
CASE WHEN ... THEN CASE ... ~1750 ~150
((((1)))) / 1+1+1+... ~1750 ~190
Nested function calls cost about six parser stack frames per level, which is worse than any other shape measured, so the real ceiling is set by the tightest case rather than the most convenient one. That worked out to roughly one hundred and ten levels in a debug build. The limit that shipped is fifty, chosen against that roomier number on purpose, because cargo test and the compiled release binary both need to survive the exact same query.
Twelve different nesting shapes are now driven through an actual socket rather than called directly in a unit test, and each one requires that a second, unrelated connection on a different socket, along with the listener itself, both remain alive afterward. That is the real bar here: not that the malicious query itself returns an error, but that every other session connected to the server never even notices anything happened.
What was refused
zql never opens a file for writing. There is no INSERT, UPDATE, DELETE, or DDL support of any kind, and this is not an unfinished corner of the project. It removes transactions, locking, and constraint checking from the problem entirely, in one decision. There is also no pg_catalog, so psql's backslash commands such as \dt are refused by name, with a message pointing the user toward SHOW SOURCES instead, rather than failing with an obscure lexer error nobody asked to see.
There is no authentication and no TLS. The protocol is plaintext only, bound to loopback by default, and the README says outright that it should not be exposed to a network you do not control.
By the numbers
Zero runtime dependencies. Seventeen substitutions against a stated bar of ten. Two hundred ninety-five tests, plus eighty-eight acceptance checks driven through a real installation of psql 16.2, plus nineteen more run through node-postgres, a completely independent client that parses results by type rather than by string, so that a wrong column type produces a visibly wrong value instead of something that merely looks plausible. Sixty-two corrupted database files, each one asserted to produce a clean error rather than a crash. Every SQLite value checked byte-for-byte against Python's own sqlite3, including i64::MIN, i64::MAX, astral-plane emoji, and a thirty-thousand-character value spanning an overflow chain. Three separate clean builds, including one produced from a fresh clone into a different directory on a different drive, all producing the same byte-identical binary.
If there is a single lesson worth taking from a project built to speak someone else's protocol using nothing but the standard library, it is that most bugs announce themselves. They panic, or they return the wrong row, or they fail a test that was already written to catch them. The one that actually cost an afternoon did none of that. It did not corrupt any data, and it did not panic. It simply used up a resource, stack space, that no test in the suite was watching for. Correctness bugs tend to get caught by testing more of the same kind of thing. This kind gets caught only by asking what your program is actually allowed to run out of, and then deliberately checking that.
Top comments (0)