I wrote this on X a few weeks ago:
I just had a very bad reminder as to the fact these LLMs are statistical parrots, I let it write code I normally wouldn't trust it to write (infra code, lots of unique behaviours) and damn
I wasn't talking about my own project when I wrote that. Then StacksNG proved me right, on its own corpus, in a hackathon I'm trying to win.
Ask my RAG assistant to verify an Interswitch webhook signature, and it didn't say "not in my knowledge base." It wrote a full authentication flow — real-looking endpoint, real-looking headers — and cited a source URL. The URL wasn't in my corpus. It wasn't anywhere. The model invented a citation for content it also invented, with zero hedging.
I'm building StacksNG for the Africa Deep Tech Challenge 2026 — an offline coding assistant scoped to the African fintech stack: Paystack, Flutterwave, Monnify, Termii. Before I submitted, I ran a 20-prompt adversarial batch against my own pipeline. Category A (in-corpus baseline) and D (phrasing brittleness) came back clean. Category B — five prompts asking about payment providers I deliberately never scraped into the corpus, Kuda, PalmPay, Interswitch, Paga, OPay — did not.
Three of five ignored a system prompt that already said, in plain language, "if the context doesn't contain enough information, say so."
That's the failure mode that zeroes out half the score in a hackathon where accuracy is 50% of the total.
My first theory was wrong, and I could prove it
My instinct was: this is a retrieval-confidence problem. Set a similarity threshold, refuse to answer below it, done.
I checked the actual numbers before writing that fix.
| Top-1 similarity | What happened | |
|---|---|---|
| Correct in-corpus answer | 0.718 | correct |
| Worst fabrication (Interswitch) | 0.712 | fully invented, fake citation |
| Correct decline (out-of-domain topic) | 0.691 | "not in my knowledge base" |
The worst hallucination had higher retrieval similarity than the cleanest correct decline. There's no threshold that lets the good case through and blocks the bad one — they're on the wrong side of each other. A confidence cutoff would have been a fix that felt right and did nothing.
What was actually happening
The chunks my retrieval pulled back for "Interswitch Quickteller" were real — Monnify's quickstart, Paystack's accept-payments guide. Genuinely similar topic: authentication, checkout, webhooks. Not out-of-domain confusion. Same-domain brand substitution. The model wasn't confused about the topic. It never checked whether the retrieved text actually named the provider I asked about, versus a different provider talking about something similar.
That's a sneakier bug than "doesn't know when it doesn't know." It's "knows something adjacent and doesn't notice the adjacency."
The fix was one paragraph, not an architecture change
I didn't retrain anything. I didn't touch retrieval. I added one rule to the system prompt:
Before answering, check whether the specific provider named in the question is actually named in the context excerpts. Retrieval is similarity-based and will sometimes hand you excerpts from a different provider just because the topic is similar — that is not the same as the named provider being covered. If it isn't named, say so. Do not substitute another provider's instructions under the asked-about provider's name, and do not invent a source URL.
Re-ran the five failing prompts plus two controls. Kuda, PalmPay, Interswitch, Paga, OPay: five for five now correctly decline. A cross-provider prompt that used to answer itself into a contradiction — "you should not fall back to X," followed immediately by a full explanation of how to fall back to X — now declines cleanly. The in-corpus control prompt is untouched, still correct.
One honest regression: my out-of-domain control prompt got slightly more hedge-y. It used to say flatly "not in my knowledge base." Now it draws an unprompted analogy to a similar provider before getting there — still no fabricated specifics, just wordier than it needs to be. Three fabrications became zero at the cost of one prompt getting less clean. I'll take that trade. I wrote the regression down instead of pretending the fix was perfect.
That was one run, though. I shipped it as the result.
An independent check found the fix wasn't a fix, it was a coin flip
I asked Antigravity (no access to my corpus authorship, my prompts, or this article) to reproduce the submission cold: fresh clone, download the model, run the official profiler, and re-test the five adversarial prompts above. Not once. Three times each, fifteen trials total.
Ten of fifteen came back clean. Not five of five. Two-thirds.
And it wasn't random noise spread evenly across providers. It split cleanly in two. Interswitch, Paga, OPay: nine for nine, 100% reliable. Kuda and PalmPay: one of three, zero of three. PalmPay fabricated a x-palmpay-signature header and a full HMAC handler on every single run. Kuda got silently rerouted to Paystack's live charge endpoint, with an invented bank code stated as fact, in two of three.
Two things were true and I'd only checked one of them. First: the chat call runs at temperature=0.2 with no fixed seed, so the same prompt doesn't reliably produce the same answer. My original "five for five" was one draw from a distribution, not a property of the fix. Second: the failure wasn't random across providers. It was concentrated exactly where retrieval is most ambiguous. Kuda and PalmPay's webhook-verification content is topically near-identical to Paystack's and Monnify's, same HMAC-SHA512 shape, same header pattern. Their retrieved chunks sit in the tightest, most confusable similarity band I measured (cosine 0.654–0.676, five chunks within 0.022 of each other). The instruction I wrote asks the model to notice when a retrieved chunk doesn't actually name the asked-about provider. It's least able to notice that exactly when the retrieved chunk is close enough to look plausible.
A soft instruction was never going to close that gap reliably, because the thing it's fighting, retrieval similarity between near-duplicate topics, doesn't go away just because I asked nicely.
The actual fix: stop asking, start checking
The corpus only covers four providers. That's a small, enumerable set. Which means the question "is this provider actually in scope" doesn't need an LLM's judgment at all. I added a deterministic gate ahead of retrieval: a list of ~25 known African fintech and banking brand names that are not in the corpus, matched by word boundary against the incoming question. Name one of them without also naming an in-corpus provider, and the question gets declined before retrieval or generation ever runs. No temperature, no seed, no chance to fabricate — just a string match.
Antigravity again, same fifteen trials, same corpus, freshly re-cloned: fifteen for fifteen, every response returned near-instantly with no LLM call at all. Regression-checked clean too. An in-corpus question still runs the full retrieval-and-generation pipeline untouched, and a genuine comparison question ("how does Kuda compare to Paystack for webhook handling?") correctly falls through to the softer instruction instead of getting blanket-refused, since that's a legitimate question the gate isn't built to answer.
It doesn't generalize. A provider I didn't think to enumerate still depends on the same soft instruction that measured 100% for three providers and 0-33% for two. That's a real limitation, not a solved problem, and it's written down as one in the repo instead of implied away.
Why this is the argument for RAG over fine-tuning, not just the reasoning
Paystack's docs are public. Flutterwave's are public. Termii's are public. A frontier lab has access to all of it, the same way it has access to Igbo and Yoruba text scattered across the public web. Access isn't the same as behavior. I made the same point on X about language before I made it about payments APIs:
Frontier labs having access to a dataset and frontier labs training on it are different things. Igbo or Yoruba text that exists publicly still gets drowned out by the sheer weight of English in pretraining. The model isn't ignorant of your language, it's just statistically indifferent to it. Representation in the data pile is not the same as representation in the model's behavior.
Paystack's docs are a rounding error in a pretraining corpus next to Stripe's. That's not a knowledge gap I can fix by asking nicer. It's why the corpus exists — RAG re-injects the drowned-out content at query time instead of hoping it survived pretraining.
I'd already decided to stay RAG-only instead of spending GPU credits on fine-tuning — the corpus is documentation, not instruction pairs, and citations matter more than they'd survive baked into weights. The hallucination bug is the evidence, not just the reasoning. I found a real correctness bug in an afternoon, and even the fix that turned out to be incomplete was a paragraph of English and, later, a list of strings — not a training run. If this had been in the weights, finding out my first attempt only worked two-thirds of the time would have meant retraining, not rereading a diff.
The number that matters
Before any of this, I had a submission that fabricates working-looking code with fake citations for three out of five providers outside its training data — in a payments assistant, where "looks right but isn't" is worse than "doesn't know."
I almost didn't re-run my control prompt after the first fix. I'd have shipped the fabrication count as zero and missed the one place it got worse instead of better. And I almost stopped there: one clean run, five for five, box checked.
All twenty of those original prompts were mine, though, written by the person who also wrote the fix, which is exactly the setup that lets a bug hide. So I went looking for a test I didn't write, twice. First, one I couldn't have written: a developer on X and a stranger on Reddit, unconnected to each other, both stuck on the same real thing: making a Paystack webhook handler idempotent. Neither had my corpus in mind. Neither had my prompts. I ran it anyway. Top retrieved chunk was Monnify, same shape of mismatch that caused the original bug. It stayed on Paystack, gave the real fix, cited only what it actually retrieved.
Second, Antigravity re-running my own adversarial set (the one I had already tested) fifteen times instead of once. That's the run that found the fix was two-thirds reliable, not fully. The fifteen-trial number is less flattering than the five-for-five I almost shipped, and it's the one that's actually true. The deterministic gate that replaced the soft instruction got checked the same way: not "does it look right," but "does it hold up when someone with no stake in the answer runs it enough times to catch the unlucky draw."
The bug wasn't boring. My first fix of it wasn't finished.
StacksNG is an entry in the Africa Deep Tech Challenge 2026. Code, corpus scrapers, and the full stress-test transcript are in the repo.
Top comments (28)
This is one of the most honest RAG write-ups I've read here - specifically because you re-ran the thing you'd already published and printed the less flattering number. "One draw from a distribution, not a property of the fix" belongs on a wall.
One thing I'd watch with the deterministic gate: you now maintain two sources of truth about scope by hand - the corpus (what's in) and the ~25-name denylist (what's out). They're disjoint today, and nothing checks they stay disjoint. The day you scrape Kuda into the corpus and forget to remove it from the list, the gate will decline questions about a provider you actually cover - and a false decline is byte-for-byte identical to a correct one. Green either way, no error message anywhere.
Two cheap moves against it: derive the in-scope provider list from the corpus at index time (the corpus is already the authority on what's covered, so let the gate read scope from it instead of from a hand-kept list). And give the gate the same discipline you gave the model: a control pair that runs on every index build - one prompt naming an in-corpus provider must pass through to retrieval, one naming a listed-out provider must decline. A string match can't be talked out of anything, but it can absolutely rot.
Honest question: did the two strangers' Paystack-idempotency test make it into the repo as a permanent regression prompt, or was it a one-time run? Found prompts are gold precisely because you couldn't have written them.
Quick follow-up, because after writing the above I went and read the gate code properly - and now I have three specific findings instead of one general worry. One of them is a false decline you can reproduce today with an innocent, fully in-scope question.
I didn't want to file them as a public issue while the challenge is still being judged - that felt like it should be your call, not mine. So: GitHub issue, or anything else, or right here once judging is over? Whichever you prefer, they're written up and ready.
Heinrich, went and tested it myself before waiting on your write-up. Found it: "How do I implement BVN verification for a customer with a Wema Bank account?" was declining, and it shouldn't, BVN verification is genuinely covered, Wema's just the customer's bank, not the API being asked about. Same pattern with GTBank and Access Bank in a virtual-account context.
Already fixed and pushed. Pulled the seven plain commercial banks (GTBank, Access, Wema, Sterling, VFD, Providus, 9PSB) off the gate, kept the actual competing providers on it. Re-verified against all 5 adversarial prompts and both registered test prompts, no regression.
Which confirms the two-sources-of-truth problem you flagged is real, I hadn't defended against it at all. corpus.db and the denylist are both hand-maintained, nothing checked they stayed disjoint, and a false decline looked byte-for-byte like a correct one.
Deriving in-scope providers from corpus.db is the easy half now, the source column's already ground truth. The denylist's harder, it's inherently a list of things NOT covered. Your control-pair test is the right tripwire for that side.
On the Paystack idempotency prompt: honest answer, one-time run, never wired into a regression suite. Real gap too.
Is what I found the same class as yours, or something else? Either way, GitHub security advisory or email works, skip the public issue for now like you said.
Same class - false decline from an over-broad gate - but a different mechanism, and the difference matters for the fix. Yours is role confusion: the name is genuinely present, but it's the customer's bank, not the API being asked about. The gate asks "does the question contain a listed name" when the real question is "which provider's API is this question about." Mine is lexical collision: a brand name that is also an ordinary English word. "How do I add a carbon copy recipient to my transactional emails?" is an in-scope Termii question naming no fintech brand - and it declines today, citing Carbon the lender. Your bank-removal fix doesn't touch that one: Carbon is a genuine competing provider, so it stays listed, and the collision stays live.
Which means the gate now has three demonstrated failure mechanisms - role confusion, lexical collision, list drift - and still zero tests. Your mutation-tested search filter is still the only guarded guard in the repo. GitHub security advisory works for me: it stays private until you publish it, and it lives where the fix lands. I'll file the full write-up there today.
And respect for "went and tested it myself before waiting on your write-up" - that's the whole discipline in one move.
Heinrich, fixed and pushed. Collocations get stripped from the question text before the word-boundary match runs, so "carbon copy" and "racing stripe" don't fire, but a bare "Carbon's loan API" still does.
tests/test_provider_gate.py is up to 8 cases: both false-decline classes, the original adversarial set, registered prompts, and a drift guard.
You're right that role confusion and lexical collision needed different fixes, one's about what the name refers to, the other's about whether it's even acting as a name. List drift's got a guard too now, the honest version: a test that fails if corpus.db's sources and the alias dict ever stop matching exactly, not the live-derivation you actually suggested. Didn't want to touch the hot path of every query 7 hours before the deadline. Safer half-measure, not the full fix, said as much in REPORT.md.
Security advisory sounds right, thanks for filing it there instead of public. Send whatever else you've got, I'd rather find failure mode four from your write-up than from a judge.
Filed is the wrong word, it turns out - GitHub tells me private vulnerability reporting isn't enabled on stacksng, so there's nowhere to put it yet. Two ways: flip it on (Settings → Advanced Security → Private vulnerability reporting) and I'll file within the hour, or DM me an address on X.
The honest content update: your three fixes just shrank my write-up. Gate tests with a drift guard covers finding 1, the exact-match corpus/denylist test covers finding 2 (and the half-measure is the right call 7 hours before a deadline - a hot-path change now would be the riskier bug), collocation stripping covers finding 3. What's left is genuinely minor, failure-mode-four material rather than headline: single-word brands in non-English phrasings ("paga" is Spanish for "pays" - a Spanish-language question with no in-corpus name still declines citing Paga), and the mirror image of your collocation fix - a question genuinely about Carbon that happens to phrase as a collocation now passes the gate into the 0-33 % soft path. Both are documented-limitation material, not must-fix-before-judging. I'll put the full list in the advisory once the door exists.
"I'd rather find failure mode four from your write-up than from a judge" - that sentence is the whole discipline. Good luck with the deadline.
Filed is the wrong word, it turns out — GitHub tells me private vulnerability reporting isn't enabled on stacksng, so there's nowhere to put it yet. Two ways: flip it on (Settings → Advanced Security → Private vulnerability reporting) and I'll file within the hour, or DM me an address on X.
The honest content update: your three fixes just shrank my write-up. Gate tests with a drift guard covers finding 1, the exact-match corpus/denylist test covers finding 2 (and the half-measure is the right call 7 hours before a deadline — a hot-path change now would be the riskier bug), collocation stripping covers finding 3. What's left is genuinely minor, failure-mode-four material rather than headline: single-word brands in non-English phrasings ("paga" is Spanish for "pays" — a Spanish-language question with no in-corpus name still declines citing Paga), and the mirror image of your collocation fix — a question genuinely about Carbon that happens to phrase as a collocation now passes the gate into the 0-33 % soft path. Both are documented-limitation material, not must-fix-before-judging. I'll put the full list in the advisory once the door exists.
"I'd rather find failure mode four from your write-up than from a judge" — that sentence is the whole discipline. Good luck with the deadline.
Heinrich, private vulnerability reporting is on now, go ahead and file whenever ready.
Appreciate the confirmation that the 3 fixes hold and specifically that the drift-guard half-measure was the right call under the deadline, that's useful to hear from someone who isn't the one who made the call. The two new ones (Spanish "paga," the collocation-fix's own blind spot for genuine Carbon mentions) sound right as documented-limitation material rather than something to chase in the time left. They'll go in REPORT.md once your advisory's up.
Filed: GHSA-2mxx-2x4f-vrh3 - historical record of the three fixed mechanisms plus the two documented-limitation items (Spanish "paga", the collocation fix's blind spot for genuine Carbon questions), phrased so you can lift them straight into REPORT.md. Impact section says what it is: correctness/availability, not classic security - filed privately at your request during judging. All fix credit is yours, explicitly. Good luck tonight.
Heinrich, advisory's in and read. Both items are staying documented, not fixed, at least for now, they're narrow enough (English-only scope already excludes the Spanish case; the Carbon bypass needs a fairly specific phrasing collision) that they're not worth risking a repo change over while Gate 1's still being judged. Appreciate you filing it properly instead of just leaving it in a comment thread.
Documented-not-fixed is a completely legitimate triage, and honestly the mature one while Gate 1 is under judging - both residuals are narrow, scoped, and now on the permanent record where the next maintainer can find them. That's exactly what the advisory mechanism is for: the alternative wasn't "fixed", it was "buried in a comment thread". Glad it's filed; ping me if either scope ever widens.
The similarity table is the part everyone building RAG should screenshot. That the worst fabrication scored higher than your cleanest correct decline is the whole argument against confidence-cutoff grounding in one line — the good and bad cases just aren't linearly separable on a single scalar.
Your prompt-level name-check fix is the right instinct, but I'd be curious how brittle it is under phrasing drift. We've had the same "adjacent, not absent" problem, and a prompt rule that says "check the provider is named in the excerpts" tends to work until the model decides "Quickteller" and "Interswitch" are close enough to count as named. What eventually held up for us was moving the check out of the prompt entirely: a cheap deterministic pass that verifies the specific entity from the query appears verbatim (or via a known alias table) in the retrieved chunks before generation runs, and forces a decline if not. Slower by one string match, but it can't be talked out of it the way the model can. Did you consider making the name-check a hard gate rather than an instruction the model can override?
Max, you called it before I even knew it was broken. I published this claiming 5/5 clean declines on one run. Then an independent re-test, 15 trials, 5 providers × 3 reps, showed it was actually 10/15, and not evenly either: three providers held 100%, but Kuda and PalmPay came back at 33% and 0%.
It wasn't really phrasing drift. It was retrieval-similarity drift, those 2 just sit in the tightest, most confusable similarity band to Paystack/Monnify in the whole corpus, so the model had every reason to talk itself into "close enough."
So I ended up building the hard gate you're describing. Word-boundary match against known out-of-corpus providers, checked before retrieval or generation even runs. 15/15 on the re-test.
It only covers providers I thought to list in advance, though. How do you handle the ones nobody thought of, does your alias table stretch to entities neither list has seen, or is that still on the model?
The “check existence before explaining” rule is a strong practical guardrail. I’d also make the verifier return an explicit evidence object—source, version, symbol, and confidence—so the UI can distinguish “not found” from “not indexed yet.” That seems especially important for fast-moving APIs where stale documentation can look like a hallucination.
Max, "not indexed yet" versus "not found" is a distinction the gate doesn't make right now. It's just binary, which is a real gap for anything that's actually missing from the corpus rather than genuinely out of scope.
Exactly—making the existence check explicit also gives you a clean place to instrument failures. I’d log which entity check failed, what evidence was retrieved, and whether the system refused or escalated, then turn those cases into regression tests. Over time that should improve reliability without encouraging the model to become more confident than the evidence allows.
The similarity-score comparison is the part that stood out to me. A relevant-looking chunk can still belong to the wrong provider, so retrieval confidence alone cannot prevent fabrication. Moving that check outside the LLM with a deterministic gate was the right call, especially for payment APIs where a convincing wrong answer is dangerous.
Paul, the payment-API framing's the right way to think about it. A wrong answer that's also unconvincing gets caught in code review anyway, it's the convincing wrong answer that actually ships.
heinrichneb found the sharper thing in this thread: a gate that converts fabrication into a new class of false refusal is not obviously progress, and the discussion has been scoring it as if it obviously were. Everyone here counts the gate the same way, how often it correctly refused against how often it fabricated, and nobody has asked how the rubric underneath weighs a refusal against a confidently wrong answer. That weighing decides whether the trade was worth making.
A practitioner who went through ATM-Bench published the breakdown, and it is blunt: none of the instruments in common use credit a refusal above a confidently wrong answer, and the best case across them is a tie at zero. On MTRAG, which does give a correct refusal full credit, the refusal-eligible slice is 55 tasks against 709 answerable ones, so getting every one of those right caps out at about 4.6 points of the suite, while a single false refusal on the answerable majority costs roughly 0.82.
Put those two numbers next to a clean 15 out of 15 and the arithmetic stops being reassuring. Three fabrications traded for a handful of new false refusals on legitimate questions could be a wash or a net loss, depending on where the refusals landed. The only way to know is checking the swap against the rubric an evaluation actually uses rather than against your own test suite, and I cannot tell which way your fifteen fall without that breakdown.
The “adjacent, not absent” failure mode is particularly important. A retrieval system can return highly relevant-looking content while still being wrong for the specific entity in the query. That makes relevance scoring alone insufficient; entity validation needs to happen somewhere between retrieval and generation. Otherwise, the model can produce a perfectly coherent answer that is grounded in real documentation—just not the documentation you actually asked for.
Glen, yeah, that's basically what the deterministic gate is built around. Grounded-but-wrong fools a reviewer way more easily than ungrounded-and-wrong does, which is why it had to get caught before generation instead of graded after the fact.
The fabricated citation is the scary part, it makes the hallucination look grounded. Abstention instructions alone rarely hold, the model still fills the gap when retrieval is thin, so a hard evidence threshold that refuses below a match score was what finally stopped it for me. Did the invented endpoints share any pattern?
Kartik, that's actually the fix I tried first and the numbers ruled it out. Worst fabrication scored 0.712 similarity. Cleanest correct decline scored 0.691. The bad case scored higher, so there's no cutoff that lets the good one through without letting the bad one through too.
I think threshold and name-check are solving two different bugs. Thin retrieval, not enough said about anything, is a similarity problem. Wrong-entity retrieval, plenty said, just about a different provider, isn't. Confidence just doesn't encode which entity got matched.
As for the pattern in the invented endpoints: it was always the exact HMAC/header shape of the nearest real provider, never anything structurally novel. The model wasn't inventing, it was relabeling.
What corpus was your threshold tuned against? Is entity substitution actually rare there or did it just never get adversarially tested??
The "invented citation for invented content" failure mode is exactly what makes RAG evals so hard to trust. Category B is where every demo looks clean until demo day.
One thing that helped on my side: I started running the same 20-prompt adversarial batch against a local model (Ollama + a 7B) as a baseline. Turns out the smaller local model actually abstained more often — worse answers, but fewer confident hallucinations. That changed my default: cloud model for generation, local model as a cheap "does this citation even exist in the corpus" fact-checker before anything reaches the user.
Did you end up penalizing invented URLs programmatically (e.g. checking every cited link against the corpus index), or was it manual review? I've been sketching an automated check for my own pipeline with an AI coding assistant and the corpus-lookup part is the piece I keep overcomplicating.
Niuniu, no programmatic URL check and here's why that wasn't actually the fix I needed. Citations in this pipeline are never LLM-generated, they're just the retrieved chunk's own Source field, printed as-is after generation. So a citation can't get invented out of nothing for an in-corpus question, it's tied to whatever row retrieval actually pulled.
Which is also why it doesn't catch everything. If retrieval grabs the wrong row (right provider, wrong section, or the adjacent-provider mess from the article), the citation's real and still wrong. The gate stops fabrication for known out-of-corpus asks by declining before generation even runs. But nothing right now checks whether an in-corpus citation actually answers what was asked.
Your local-model fact-checker sounds like it's solving this from the other end though. Does it check that the specific entity in the citation matches the query, or just that the URL exists somewhere in the corpus? The second one wouldn't have caught my Interswitch-cites-Monnify bug, Monnify's URL is real after all.
The entity-check is a great distinction from a generic retrieval-confidence threshold. I’d be curious to see this extended into an automated invariant: whenever the answer contains a provider, endpoint, or product name, verify that it appears in the retrieved evidence before allowing the response through. That kind of post-generation check seems especially valuable for fintech, where a plausible cross-provider substitution can be much worse than a refusal.
The 0.712 vs 0.691 table is the part I'd quote back at anyone proposing a similarity threshold. Retrieval scores measure topic adjacency, not entity coverage, so they can't separate your worst fabrication from your cleanest decline, they're literally on the wrong side of each other.
One thing that helped in a similar setup: pull the entity check out of the system prompt and make it deterministic. Before generation, match the provider name from the question against the retrieved chunks; zero hits means decline, no model call. The prompt rule you added still asks the model to police itself, and your own independent reproduction showed how that goes. A substring check fails closed every time, and it's free to run.
Did you end up keeping the check in the prompt, or did it move into the pipeline after the coin-flip result?