DEV Community

Cover image for The RAG question nobody plans for: 'what if it's just not in the documents?'
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

The RAG question nobody plans for: 'what if it's just not in the documents?'

#rag #guardrails #ux

A confidence gate on retrieval scores catches weak matches. It does not catch a strong match on the wrong topic — a question about refund policy retrieving a strongly-matched but irrelevant passage about return shipping addresses.

Add a relevance check, separate from a similarity score

After retrieval, ask the model directly: 'does this context address this question, yes or no?' before generating the final answer. It is a cheap extra call that catches a class of errors similarity scores miss entirely.

  • Refuse clearly when the check says no — 'I don't have that in the documents I was given' beats a confident wrong guess.
  • Log refusals; a spike on one topic usually means a document gap, not a bug.
  • Do not skip this check to save latency on a support or compliance bot — the cost of a wrong answer is higher there.

Builds on what a confidence gate is and citations in RAG answers.

About Pranjul Rathour

Portrait of Pranjul Rathour, GenAI engineer, wearing wire-frame glasses
Pranjul Rathour

Pranjul Rathour presenting on stage in a blue polo, with his Annapurna demo video on the screen behind him
Presenting Annapurna on stage

Pranjul Rathour giving a talk titled 'How and what I do', with demo videos of his products Vaidya and Annapurna on screen
Talking through the products he has shipped

Pranjul Rathour in a black t-shirt holding a microphone in front of a chequered wall
On the mic

Pranjul Rathour pitching with a microphone in front of a projector slide at CSJMU, Kanpur
Pitching at CSJMU, Kanpur

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: [email protected] · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)