RAG Eval

Your RAG system works. Probably. You can’t currently prove it.

The demo went well. Then someone in the room typed a question you hadn’t tried, got a confidently wrong answer with a citation attached, and now you’re being asked whether the whole thing is broken.

You need a number, and you don’t have one — because “is our RAG good” isn’t a question a vibe check can answer, and the metrics that look like answers mostly aren’t.

This site is about measurement. Not what RAG is. Whether yours works.

What you’ll find here

• Building an eval set that isn’t rigged. The most common failure is questions written by someone who already knew which document held the answer. Those score beautifully and predict nothing.
• Retrieval metrics and their blind spots. Recall@k, MRR, nDCG — what each one is sensitive to, and the specific ways a high score coexists with an unusable system.
• Faithfulness, separately from quality. An answer can be well-written, helpful, and unsupported by anything you retrieved. Those are different measurements and conflating them is how bad systems pass review.
• LLM-as-judge, with the caveats attached. It’s the only scalable option for open-ended evaluation and it has real, documented biases. Both facts at once.
• Which stage broke. Parsing, chunking, embedding, retrieval, reranking, prompt assembly, generation. A bad answer implicates one of seven things, and there’s a fast way to find out which.

The discipline this site keeps: retrieval metrics, generation metrics, and end-to-end metrics are three different things. Nearly every confused RAG evaluation is confused because someone averaged across that boundary.

Written for people who already shipped something. Code where it helps, none where it doesn’t.

Read the blog →

No benchmark numbers presented as universal. Whether reranking helps your corpus is an empirical question about your corpus, and anyone quoting you a percentage is quoting someone else’s data.