Evaluating RAG With Ragas: A Hands-On Guide

Ragas (short for RAG Assessment) is an open-source Python framework for evaluating retrieval-augmented generation pipelines, published by Shahul Es and colleagues and presented at EACL 2024. It uses a judge language model to score your pipeline on groundedness and relevance, so you can tell whether a bad answer came from broken retrieval or a hallucinating generator.

Key takeaways

• Text-overlap scores like BLEU and ROUGE cannot judge RAG: they punish correct paraphrases and miss hallucinations. LLM-as-judge metrics are the 2026 standard.
• Ragas scores two failure surfaces separately: faithfulness plus answer relevancy diagnose the generator, while context precision plus context recall diagnose the retriever.
• On Ragas 0.4.3 you import metric classes from ragas.metrics.collections, wrap a judge model with llm_factory(), and build datasets from SingleTurnSample objects. Old v0.1 and v0.2 tutorials will not run.
• Sensible starting thresholds are faithfulness above 0.80, answer relevancy above 0.85, context precision above 0.75 and context recall above 0.80. Calibrate from your own baseline, not from a blog post.
• Scores are estimates, not ground truth: they inherit the judge model's biases and drift when the judge changes. Keep the judge fixed and re-baseline on change.

Why BLEU and ROUGE are the wrong ruler for RAG

Traditional NLP metrics compare generated text to a reference answer token by token, which fails twice for RAG. A correct answer phrased differently scores badly, while a fluent answer that invents facts can score well, which is exactly the failure RAG exists to prevent.

Ragas takes a different route: it asks a judge language model to score each answer along four axes. Two axes check the generator (is the answer grounded in what was retrieved, and does it address the question?), and two check the retriever (were the right chunks fetched, and were they ranked well?). This matters at scale: according to LangChain's State of AI 2026 report, 62 per cent of production LLM applications now include retrieval augmentation.

What are the four Ragas metrics?

Written against Ragas 0.4.3, the current release as of October 2026. The API changed substantially between 0.1, 0.2 and 0.4, so treat older tutorials as suspect.

Faithfulness: is the answer grounded?

Faithfulness, also called groundedness, checks whether every claim in the generated answer is supported by the retrieved context. Ragas splits the answer into atomic statements and verifies each one against the chunks, returning a score between 0 and 1. A faithfulness of 0.6 means roughly 40 per cent of the statements have no basis in the retrieved material, which is hallucination in the strict sense. Most production teams require 0.80 or higher, and regulated domains such as finance or healthcare should demand 0.90.

Answer relevancy: does it address the question?

Answer relevancy measures whether the response actually answers what was asked, independent of whether the facts are right. It combines a judge LLM with embedding similarity. This metric catches a common failure: an answer that is perfectly grounded in the context but answers a different question, which happens when prompts drift or the generator latches onto the wrong chunk.

Context precision: were the chunks ranked well?

Context precision checks that the relevant chunks appear near the top of the retrieval ranking. Unlike the first two metrics, it needs a ground-truth reference answer to decide which chunks count as relevant. Use it when you suspect the retriever finds the right documents but buries them below noise.

Context recall: was anything missed?

Context recall measures whether the retriever fetched all the information needed to answer the question, again against a ground-truth reference. Low recall with high precision is a classic chunking problem: your chunks are good, but the split cut the answer across boundaries the retriever cannot bridge.

Which metrics should you run first?

If you have no reference answers yet, start with faithfulness and answer relevancy. Both are reference-free: they need only the question, the retrieved contexts and the generated answer. They catch the failures users actually feel, which are hallucinated or off-topic answers. Once you have a golden set of question and reference-answer pairs, add context precision and context recall to start tuning the retriever.

How do you evaluate a RAG pipeline with Ragas?

Install the current release and set a judge model key:

pip install ragas
export OPENAI_API_KEY=sk-your-key-here

The cheapest serious judge is a small model such as gpt-4o-mini or an OpenAI-compatible local model; it needs to follow evaluation instructions reliably, not to write poetry. Next, run your own pipeline over a set of test questions and capture what it retrieved and generated. Each result becomes one SingleTurnSample:

from ragas import SingleTurnSample, EvaluationDataset

samples = [
    SingleTurnSample(
        user_input="Which port does the SettlePost API use in production?",
        retrieved_contexts=[
            "The SettlePost production API listens on port 8443 behind the load balancer.",
            "Staging uses port 8080. Development servers run on 3000.",
        ],
        response="The SettlePost production API runs on port 8443.",
        reference="The production API runs on port 8443.",
    ),
    # ... one sample per test question, ideally 50 to 100
]

dataset = EvaluationDataset(samples=samples)

Then pick metrics and a judge, and run the evaluation:

from ragas import evaluate
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    Faithfulness,
    AnswerRelevancy,
    ContextPrecisionWithReference,
    ContextRecall,
)

judge = llm_factory("gpt-4o-mini")

result = evaluate(
    dataset,
    metrics=[
        Faithfulness(llm=judge),
        AnswerRelevancy(llm=judge),
        ContextPrecisionWithReference(llm=judge),
        ContextRecall(llm=judge),
    ],
)

print(result)
# {'faithfulness': 0.93, 'answer_relevancy': 0.91,
#  'context_precision': 0.83, 'context_recall': 0.79}

For debugging one bad answer, every metric exposes an async single_turn_ascore() method that takes a single SingleTurnSample.

How do you read the scores like a diagnosis?

The point of the four-metric split is localisation: the pattern of failures tells you which component to fix. Use this table as a starting point.

What you seeWhat it meansWhat to tryLow faithfulness, high context recallThe retriever found the facts, but the generator ignored themTighten the system prompt to cite context, lower temperature, or switch to a more instruction-following generatorLow faithfulness, low context recallThe generator had nothing good to work withFix retrieval first: this is not a prompting problemHigh faithfulness, low answer relevancyGrounded answer to the wrong questionCheck prompt drift, rewrite ambiguous test questions, inspect which chunk the generator latched ontoLow context precision, high context recallRight documents, wrong rankingAdd a cross-encoder reranker or tune hybrid search weightsLow context recallThe retriever never found the needed chunkRevisit chunking strategy and chunk size, expand top-k, add BM25 hybrid search

What does a Ragas score not tell you?

Honesty first: these scores are estimates produced by a model, not measurements against ground truth. Three caveats matter in practice.

First, judge bias. A judge model tends to favour answers written in its own style and can be fooled by confident phrasing, so a rising faithfulness score after swapping the judge may be flattery, not progress. Keep the judge fixed across runs and re-baseline your thresholds whenever you change it.

Second, cost and latency. Every metric on every sample is one or more LLM calls, plus embedding calls for the relevancy metrics. A hundred-question suite with four metrics is cheap; a ten-thousand-question regression suite with a frontier judge is a budget line. Most teams run a small golden set on every release and a larger sample nightly.

Third, test data quality. Ragas ships a TestsetGenerator that can synthesise question and answer pairs from your documents, which is a fast start, but synthetic questions mirror the generator's assumptions. Treat synthetic sets as scaffolding and grow a set of real user queries over time.

Can you gate releases on Ragas scores?

Yes, and this is where evaluation pays for itself. Store the scores from your golden set as a baseline and fail the build when any metric drops beyond an agreed margin. A minimal CI step looks like this:

# run this after building your RAG service, before deploy
python -m pytest tests/test_rag_eval.py -q

# inside test_rag_eval.py
def test_rag_quality_gate():
    result = evaluate(dataset, metrics=[...])  # as above
    assert result["faithfulness"] > 0.80, f"Faithfulness regressed: {result['faithfulness']}"
    assert result["answer_relevancy"] > 0.85
    assert result["context_precision"] > 0.75
    assert result["context_recall"] > 0.80

Thresholds deserve one warning: pick them from your own baseline, not from this article. A pipeline answering simple lookups can hold 0.95 faithfulness easily; one doing multi-hop reasoning over messy PDFs may stabilise at 0.82. What matters is the trend, and the gate catches it before your users do.

Evaluation is the difference between a RAG demo and a RAG product. Demos get judged by a vibe check on five hand-picked questions; products get judged by thousands of users on questions you never imagined. Ragas gives you the vocabulary and the tooling to close that gap: build the dataset, run the four metrics, read them diagnostically, and put the gate in CI.

Sources: Ragas documentation (docs.ragas.io); the DataCamp tutorial on RAG evaluation with Ragas 0.4.3; aiworkshack's hands-on guide to Ragas metrics on 0.4.x; the dev.to walkthrough of the four core metrics; FutureAGI's 2026 survey of LLM evaluation frameworks; and the Ragas paper by Shahul Es and colleagues, presented at EACL 2024.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.