Every security tool has a dirty secret: nobody tests whether the detector detects. The tests pass, the CI is green, and everyone moves on. Today I ...
For further actions, you may consider blocking this person and/or reporting abuse
@peterbuildssecure You just named fixture 3, and it goes on the list today. The part that makes it nastier than the zero-width case: zero-width stripping and NFKC both normalize a copy of the text, but the homoglyph survives as a perfectly legal, visible character. My flag heuristic (normalized copy trips, original does not) still works as the detector shape, but the normalization table has to come from a source with authority, and Unicode confusables.txt is exactly that source.
The trap I want to avoid is shipping a half table (Cyrillic a, Greek o) and calling it done. The confusables list is long and updated on revision. So the plan: fixture lands first, table gets pulled from source with the revision pinned, and the scanner reports that revision in its output so results stay reproducible. Thanks for closing the gap I only half saw.
Worth pinning the Unicode Character Database (UCD) version too, alongside the confusables.txt revision. NFKC normalization itself isn't stable across UCD versions -- new codepoints, changed decompositions -- so the same input string can normalize differently depending on which UCD your normalization library shipped with. If the scanner reports the confusables.txt revision but not the UCD version behind its NFKC step, two runs on different library versions can silently disagree on whether a given string trips the heuristic, even reading the same confusables table. Pin both, report both in the output.
Good catch, and it's the same reproducibility problem one layer down. I was treating confusables.txt as the only moving part, but if NFKC runs on whatever Unicode data ships with the Python build, two machines can disagree without either one being wrong. Python exposes that as unicodedata.unidata_version, so reporting it next to the confusables revision should be cheap. "Pin both, report both" goes into the fixture 3 plan as you wrote it. Do you know whether confusables.txt revisions are tied to a specific UCD version, or can the two drift apart?
They're published together in each Unicode release — confusables.txt is part of UTS #39, versioned alongside the UCD for that release — but that doesn't stop them drifting in practice, because they usually don't come from the same place at runtime. Python's unicodedata module bundles whatever UCD version shipped with that interpreter build, but if your fixture pulls confusables.txt fresh from unicode.org (or vendors it separately), you can end up pairing, say, UCD 15.0's unidata_version with a confusables.txt from 15.1. Same failure mode as the original bug: two 'authoritative' sources that happen to usually agree, until a release cycle where they don't. If you're pinning both, worth pinning them as a pair from the same Unicode release rather than independently — pin-drift between correctly-pinned-but-mismatched versions is the same silent divergence as not pinning at all.
"Correctly pinned but mismatched is the same as not pinned" is the useful rule here. I'll pin unidata_version and confusables.txt as a pair from the same Unicode release, and have the scanner refuse to run (or at least warn loudly) if the interpreter's UCD doesn't match the vendored confusables version. Thanks for walking this through two layers, the fixture plan is much better for it.
@reidmarlow Sharpest framing of the miss I have read, and re-checking the AST after your comment added a nuance: Python folds adjacent string literals into a single Constant at parse time, so a pure AST visitor would already see the joined text. My scanner missed it because the rule scans raw source lines, not the AST: a phrase split across lines never appears whole in any single line, and no folding happens at the line-scan layer. Your lexer-level token join and my pre-join fix converge on the same rule: join before match.
The case where your literal_eval pass earns its keep is explicit concatenation, a BinOp like "foo" + "bar", where there is no adjacency for the parser to fold. That one goes on the fixture list next to the homoglyph case. Two new blind spots found in one comment thread, this is exactly why I ship the misses.
The concatenated string split is brutal for regex scanners. As soon as code authors or LLMs format long prompts as adjacent string literals across multiple lines, standard line-based AST visitors treat them as separate constant nodes unless you explicitly run a constant-folding pass.
AST visitors that inspect ast.Constant directly without constant folding miss this every time. A quick pass with ast.literal_eval on joined string expressions or tracking adjacent string tokens at the lexer level makes that fixture trivial to catch before touching full semantic analysis.
That matches the fix direction I wrote down, pre-joining adjacent string literals before matching, but you've given it a more concrete home. Working at the token level appeals to me because it skips a full AST pass and keeps the original line numbers for the finding. Would you fold only implicit concatenation, or also explicit + between two literals?
The homoglyph case is worth adding as a third obfuscation fixture, since it defeats both of your current normalization ideas differently than zero-width chars do. Swap a Latin letter in the trigger phrase for a visually identical Cyrillic or Greek one (a→а, o→ο) and the literal pattern misses for the same reason as the zero-width case, but stripping zero-width/bidi characters won't fix it — the character is present and rendered, just not the codepoint the regex expects. The general fix is NFKC-normalizing plus a confusables table (Unicode's own confusables.txt) applied to a canonicalized copy before matching — same "flag when canonical trips but original doesn't" pattern you're already using for the zero-width case, just a different normalization step. Worth adding now while you're already building the obfuscation fixture set.
Homoglyphs are a better third fixture than anything I had planned, because they break both of my normalization ideas in different ways. Stripping zero-width and bidi characters does nothing when the character is visible and just has the wrong codepoint. NFKC + a confusables skeleton on a canonical copy, then "flag when canonical trips but original doesn't", fits the pattern I already use, so it should slot in cleanly. Adding a→а and o→ο as the first two cases.