I wrote this on X a few weeks ago:
I just had a very bad reminder as to the fact these LLMs are statistical parrots, I let it write code I normal...
For further actions, you may consider blocking this person and/or reporting abuse
This is one of the most honest RAG write-ups I've read here - specifically because you re-ran the thing you'd already published and printed the less flattering number. "One draw from a distribution, not a property of the fix" belongs on a wall.
One thing I'd watch with the deterministic gate: you now maintain two sources of truth about scope by hand - the corpus (what's in) and the ~25-name denylist (what's out). They're disjoint today, and nothing checks they stay disjoint. The day you scrape Kuda into the corpus and forget to remove it from the list, the gate will decline questions about a provider you actually cover - and a false decline is byte-for-byte identical to a correct one. Green either way, no error message anywhere.
Two cheap moves against it: derive the in-scope provider list from the corpus at index time (the corpus is already the authority on what's covered, so let the gate read scope from it instead of from a hand-kept list). And give the gate the same discipline you gave the model: a control pair that runs on every index build - one prompt naming an in-corpus provider must pass through to retrieval, one naming a listed-out provider must decline. A string match can't be talked out of anything, but it can absolutely rot.
Honest question: did the two strangers' Paystack-idempotency test make it into the repo as a permanent regression prompt, or was it a one-time run? Found prompts are gold precisely because you couldn't have written them.
Quick follow-up, because after writing the above I went and read the gate code properly - and now I have three specific findings instead of one general worry. One of them is a false decline you can reproduce today with an innocent, fully in-scope question.
I didn't want to file them as a public issue while the challenge is still being judged - that felt like it should be your call, not mine. So: GitHub issue, or anything else, or right here once judging is over? Whichever you prefer, they're written up and ready.
Heinrich, went and tested it myself before waiting on your write-up. Found it: "How do I implement BVN verification for a customer with a Wema Bank account?" was declining, and it shouldn't, BVN verification is genuinely covered, Wema's just the customer's bank, not the API being asked about. Same pattern with GTBank and Access Bank in a virtual-account context.
Already fixed and pushed. Pulled the seven plain commercial banks (GTBank, Access, Wema, Sterling, VFD, Providus, 9PSB) off the gate, kept the actual competing providers on it. Re-verified against all 5 adversarial prompts and both registered test prompts, no regression.
Which confirms the two-sources-of-truth problem you flagged is real, I hadn't defended against it at all. corpus.db and the denylist are both hand-maintained, nothing checked they stayed disjoint, and a false decline looked byte-for-byte like a correct one.
Deriving in-scope providers from corpus.db is the easy half now, the source column's already ground truth. The denylist's harder, it's inherently a list of things NOT covered. Your control-pair test is the right tripwire for that side.
On the Paystack idempotency prompt: honest answer, one-time run, never wired into a regression suite. Real gap too.
Is what I found the same class as yours, or something else? Either way, GitHub security advisory or email works, skip the public issue for now like you said.
Same class - false decline from an over-broad gate - but a different mechanism, and the difference matters for the fix. Yours is role confusion: the name is genuinely present, but it's the customer's bank, not the API being asked about. The gate asks "does the question contain a listed name" when the real question is "which provider's API is this question about." Mine is lexical collision: a brand name that is also an ordinary English word. "How do I add a carbon copy recipient to my transactional emails?" is an in-scope Termii question naming no fintech brand - and it declines today, citing Carbon the lender. Your bank-removal fix doesn't touch that one: Carbon is a genuine competing provider, so it stays listed, and the collision stays live.
Which means the gate now has three demonstrated failure mechanisms - role confusion, lexical collision, list drift - and still zero tests. Your mutation-tested search filter is still the only guarded guard in the repo. GitHub security advisory works for me: it stays private until you publish it, and it lives where the fix lands. I'll file the full write-up there today.
And respect for "went and tested it myself before waiting on your write-up" - that's the whole discipline in one move.
Heinrich, fixed and pushed. Collocations get stripped from the question text before the word-boundary match runs, so "carbon copy" and "racing stripe" don't fire, but a bare "Carbon's loan API" still does.
tests/test_provider_gate.py is up to 8 cases: both false-decline classes, the original adversarial set, registered prompts, and a drift guard.
You're right that role confusion and lexical collision needed different fixes, one's about what the name refers to, the other's about whether it's even acting as a name. List drift's got a guard too now, the honest version: a test that fails if corpus.db's sources and the alias dict ever stop matching exactly, not the live-derivation you actually suggested. Didn't want to touch the hot path of every query 7 hours before the deadline. Safer half-measure, not the full fix, said as much in REPORT.md.
Security advisory sounds right, thanks for filing it there instead of public. Send whatever else you've got, I'd rather find failure mode four from your write-up than from a judge.
Filed is the wrong word, it turns out - GitHub tells me private vulnerability reporting isn't enabled on stacksng, so there's nowhere to put it yet. Two ways: flip it on (Settings → Advanced Security → Private vulnerability reporting) and I'll file within the hour, or DM me an address on X.
The honest content update: your three fixes just shrank my write-up. Gate tests with a drift guard covers finding 1, the exact-match corpus/denylist test covers finding 2 (and the half-measure is the right call 7 hours before a deadline - a hot-path change now would be the riskier bug), collocation stripping covers finding 3. What's left is genuinely minor, failure-mode-four material rather than headline: single-word brands in non-English phrasings ("paga" is Spanish for "pays" - a Spanish-language question with no in-corpus name still declines citing Paga), and the mirror image of your collocation fix - a question genuinely about Carbon that happens to phrase as a collocation now passes the gate into the 0-33 % soft path. Both are documented-limitation material, not must-fix-before-judging. I'll put the full list in the advisory once the door exists.
"I'd rather find failure mode four from your write-up than from a judge" - that sentence is the whole discipline. Good luck with the deadline.
The similarity table is the part everyone building RAG should screenshot. That the worst fabrication scored higher than your cleanest correct decline is the whole argument against confidence-cutoff grounding in one line — the good and bad cases just aren't linearly separable on a single scalar.
Your prompt-level name-check fix is the right instinct, but I'd be curious how brittle it is under phrasing drift. We've had the same "adjacent, not absent" problem, and a prompt rule that says "check the provider is named in the excerpts" tends to work until the model decides "Quickteller" and "Interswitch" are close enough to count as named. What eventually held up for us was moving the check out of the prompt entirely: a cheap deterministic pass that verifies the specific entity from the query appears verbatim (or via a known alias table) in the retrieved chunks before generation runs, and forces a decline if not. Slower by one string match, but it can't be talked out of it the way the model can. Did you consider making the name-check a hard gate rather than an instruction the model can override?
Max, you called it before I even knew it was broken. I published this claiming 5/5 clean declines on one run. Then an independent re-test, 15 trials, 5 providers × 3 reps, showed it was actually 10/15, and not evenly either: three providers held 100%, but Kuda and PalmPay came back at 33% and 0%.
It wasn't really phrasing drift. It was retrieval-similarity drift, those 2 just sit in the tightest, most confusable similarity band to Paystack/Monnify in the whole corpus, so the model had every reason to talk itself into "close enough."
So I ended up building the hard gate you're describing. Word-boundary match against known out-of-corpus providers, checked before retrieval or generation even runs. 15/15 on the re-test.
It only covers providers I thought to list in advance, though. How do you handle the ones nobody thought of, does your alias table stretch to entities neither list has seen, or is that still on the model?
The “check existence before explaining” rule is a strong practical guardrail. I’d also make the verifier return an explicit evidence object—source, version, symbol, and confidence—so the UI can distinguish “not found” from “not indexed yet.” That seems especially important for fast-moving APIs where stale documentation can look like a hallucination.
Max, "not indexed yet" versus "not found" is a distinction the gate doesn't make right now. It's just binary, which is a real gap for anything that's actually missing from the corpus rather than genuinely out of scope.
Exactly—making the existence check explicit also gives you a clean place to instrument failures. I’d log which entity check failed, what evidence was retrieved, and whether the system refused or escalated, then turn those cases into regression tests. Over time that should improve reliability without encouraging the model to become more confident than the evidence allows.
The similarity-score comparison is the part that stood out to me. A relevant-looking chunk can still belong to the wrong provider, so retrieval confidence alone cannot prevent fabrication. Moving that check outside the LLM with a deterministic gate was the right call, especially for payment APIs where a convincing wrong answer is dangerous.
Paul, the payment-API framing's the right way to think about it. A wrong answer that's also unconvincing gets caught in code review anyway, it's the convincing wrong answer that actually ships.
heinrichneb found the sharper thing in this thread: a gate that converts fabrication into a new class of false refusal is not obviously progress, and the discussion has been scoring it as if it obviously were. Everyone here counts the gate the same way, how often it correctly refused against how often it fabricated, and nobody has asked how the rubric underneath weighs a refusal against a confidently wrong answer. That weighing decides whether the trade was worth making.
A practitioner who went through ATM-Bench published the breakdown, and it is blunt: none of the instruments in common use credit a refusal above a confidently wrong answer, and the best case across them is a tie at zero. On MTRAG, which does give a correct refusal full credit, the refusal-eligible slice is 55 tasks against 709 answerable ones, so getting every one of those right caps out at about 4.6 points of the suite, while a single false refusal on the answerable majority costs roughly 0.82.
Put those two numbers next to a clean 15 out of 15 and the arithmetic stops being reassuring. Three fabrications traded for a handful of new false refusals on legitimate questions could be a wash or a net loss, depending on where the refusals landed. The only way to know is checking the swap against the rubric an evaluation actually uses rather than against your own test suite, and I cannot tell which way your fifteen fall without that breakdown.
The “adjacent, not absent” failure mode is particularly important. A retrieval system can return highly relevant-looking content while still being wrong for the specific entity in the query. That makes relevance scoring alone insufficient; entity validation needs to happen somewhere between retrieval and generation. Otherwise, the model can produce a perfectly coherent answer that is grounded in real documentation—just not the documentation you actually asked for.
Glen, yeah, that's basically what the deterministic gate is built around. Grounded-but-wrong fools a reviewer way more easily than ungrounded-and-wrong does, which is why it had to get caught before generation instead of graded after the fact.
The fabricated citation is the scary part, it makes the hallucination look grounded. Abstention instructions alone rarely hold, the model still fills the gap when retrieval is thin, so a hard evidence threshold that refuses below a match score was what finally stopped it for me. Did the invented endpoints share any pattern?
Kartik, that's actually the fix I tried first and the numbers ruled it out. Worst fabrication scored 0.712 similarity. Cleanest correct decline scored 0.691. The bad case scored higher, so there's no cutoff that lets the good one through without letting the bad one through too.
I think threshold and name-check are solving two different bugs. Thin retrieval, not enough said about anything, is a similarity problem. Wrong-entity retrieval, plenty said, just about a different provider, isn't. Confidence just doesn't encode which entity got matched.
As for the pattern in the invented endpoints: it was always the exact HMAC/header shape of the nearest real provider, never anything structurally novel. The model wasn't inventing, it was relabeling.
What corpus was your threshold tuned against? Is entity substitution actually rare there or did it just never get adversarially tested??
The "invented citation for invented content" failure mode is exactly what makes RAG evals so hard to trust. Category B is where every demo looks clean until demo day.
One thing that helped on my side: I started running the same 20-prompt adversarial batch against a local model (Ollama + a 7B) as a baseline. Turns out the smaller local model actually abstained more often — worse answers, but fewer confident hallucinations. That changed my default: cloud model for generation, local model as a cheap "does this citation even exist in the corpus" fact-checker before anything reaches the user.
Did you end up penalizing invented URLs programmatically (e.g. checking every cited link against the corpus index), or was it manual review? I've been sketching an automated check for my own pipeline with an AI coding assistant and the corpus-lookup part is the piece I keep overcomplicating.
Niuniu, no programmatic URL check and here's why that wasn't actually the fix I needed. Citations in this pipeline are never LLM-generated, they're just the retrieved chunk's own Source field, printed as-is after generation. So a citation can't get invented out of nothing for an in-corpus question, it's tied to whatever row retrieval actually pulled.
Which is also why it doesn't catch everything. If retrieval grabs the wrong row (right provider, wrong section, or the adjacent-provider mess from the article), the citation's real and still wrong. The gate stops fabrication for known out-of-corpus asks by declining before generation even runs. But nothing right now checks whether an in-corpus citation actually answers what was asked.
Your local-model fact-checker sounds like it's solving this from the other end though. Does it check that the specific entity in the citation matches the query, or just that the URL exists somewhere in the corpus? The second one wouldn't have caught my Interswitch-cites-Monnify bug, Monnify's URL is real after all.
The entity-check is a great distinction from a generic retrieval-confidence threshold. I’d be curious to see this extended into an automated invariant: whenever the answer contains a provider, endpoint, or product name, verify that it appears in the retrieved evidence before allowing the response through. That kind of post-generation check seems especially valuable for fintech, where a plausible cross-provider substitution can be much worse than a refusal.
The 0.712 vs 0.691 table is the part I'd quote back at anyone proposing a similarity threshold. Retrieval scores measure topic adjacency, not entity coverage, so they can't separate your worst fabrication from your cleanest decline, they're literally on the wrong side of each other.
One thing that helped in a similar setup: pull the entity check out of the system prompt and make it deterministic. Before generation, match the provider name from the question against the retrieved chunks; zero hits means decline, no model call. The prompt rule you added still asks the model to police itself, and your own independent reproduction showed how that goes. A substring check fails closed every time, and it's free to run.
Did you end up keeping the check in the prompt, or did it move into the pipeline after the coin-flip result?
Glad the deterministic gate ended up being a string match instead of another training cycle nice case for RAG over fine-tuning that isn't just theoretical.