Hey builders
RAGBench just hit Hugging Face.
It's an open evaluation framework for RAG systems, built for developers, with the goal of making RAG evaluation more practical and reproducible.
Instead of guessing whether your chunking strategy actually works, test it. Compare it. Measure it.
I built RAGBench to make it easier to evaluate different RAG approaches using the same documents, queries, and evaluation metrics.
So far, I've focused on comparing semantic, document-level, and parent-child chunking and measuring how they affect retrieval and answer quality.
The goal is simple:
Stop guessing. Start measuring.
RAGBench is:
• Open source
• Open data
• Reproducible
• Built for community feedback
Try the interactive benchmark:
https://huggingface.co/spaces/Gul55555/ragbench
Contributions and feedback are welcome.
I'm especially interested in:
• What RAG evaluation metrics do you actually use?
• What chunking strategies have worked best for you?
• What problems have you encountered when evaluating RAG?
• What should I add to RAGBench next?
What's broken in your RAG pipeline?
I'd love to hear what you're working on.
Top comments (2)
Comparing semantic, document-level and parent-child chunking on a fixed corpus and query set is the right axis to start with, because chunking is where most of the variance lives and it is the stage people tune last. Two things I would want from a benchmark like this. First, report retrieval and answer quality separately per strategy - parent-child usually wins on answer quality while looking mediocre on raw chunk-level recall, and a combined score hides that. Second, report the cost side next to the quality side: parent-child sends far more tokens to the model per query, so a small quality gain can be a large bill, and the honest comparison is quality per token rather than quality alone. On what breaks in practice: documents whose meaning depends on layout - tables and multi-column PDFs - where every strategy silently produces chunks that read fine and mean nothing, so a benchmark that includes a couple of those is more informative than adding another clean-prose dataset.
Really appreciate this, especially the point about separating retrieval quality, answer quality, and cost.
Right now RAGBench focuses primarily on retrieval metrics like Hit@1, Hit@3, and MRR, with LLM-based faithfulness and answer relevancy alongside them. Your suggestion to report those dimensions separately makes a lot of sense, because a strategy can look mediocre at chunk-level recall while still producing better final answers.
The “quality per token” angle is also something I hadn’t made explicit enough. I think adding token usage/cost alongside the quality metrics would make the comparisons much more practical.
And the layout-dependent documents point is excellent. Tables and multi-column PDFs are exactly the kind of real-world failure cases that a clean prose benchmark can miss. I’ll look at adding those to the corpus rather than only expanding the number of clean-text examples.
Thanks for the thoughtful feedback. This gives me some concrete directions for the next iteration of RAGBench.