The ticket said the nightly export had failed. What had actually happened was that a single customer record containing a name with an acute accent and an apostrophe had propagated through four systems, and each one had mangled it differently, and the combination produced three separate failures that nobody connected for most of a day.
The web form accepted the name correctly and stored it correctly, because that application had been written recently by people who thought about encoding. The first downstream consumer read it over a database connection that had never had its character set configured, so it received replacement characters and stored those, permanently. The second consumer received it correctly but wrote it into a fixed-width export file, where a multi-byte character silently consumed two of the positions allocated to the field and shifted everything after it by one column, which the receiving bank's parser rejected with a message about an invalid amount. The third consumer passed the name into a shell command without quoting, and the apostrophe did what apostrophes do.
What struck me afterwards was how each team's initial diagnosis was correct and useless. The bank said our amounts were wrong. The reporting team said their names were corrupted. The integration team said their job had crashed. Three tickets, three root causes, one record.
The fixes were individually boring: set the character set explicitly on every connection and stop relying on server defaults, count bytes rather than characters when the format is byte-oriented, never build a command line by string concatenation. The durable change was different. We built a small set of deliberately hostile test records and pushed them through every interface we own on a schedule. Accented characters, an apostrophe, a name longer than any sensible field, a right-to-left script, an emoji, leading and trailing whitespace. They live in the test data permanently and they catch things.
The underlying point is that our pipelines had been validated against the data we happened to have, not against the data a human being is allowed to have. A person with an accent in their name is not an edge case. They are a normal customer who happens to be the first one to walk through the part of your system nobody tested.
– Serguey Shinder
Top comments (0)