Why Your RAG Demo Works and Production Does Not

The prototype answered every question you threw at it. Two weeks after launch, the support channel has a screenshot of a confident, cited, wrong answer, and someone senior is asking whether the technology works.

The technology works. The demo was an easier problem than production in five specific, enumerable ways — and all five can be measured before launch, which is the point of this post.

1. You wrote the questions with the document open

This is the big one.

When you built the demo you opened a document, read a paragraph, and typed a question about it. That question inherited the document’s vocabulary, its framing, and its level of specificity. Retrieval then had to match text against a question derived from that text, which is close to the easiest retrieval task that exists.

Real users have never seen the document. They ask using their own vocabulary — the internal nickname for the product, the abbreviation your documentation spells out, the term the industry used before your style guide changed. Embedding models handle some of that gap and not all of it.

How to measure it: take fifty real user queries from logs — or, if you’re pre-launch, have someone who has never read the corpus write the questions from a description of what the system is supposed to do. Score retrieval on those separately from your original set. The delta between the two numbers is the size of your rigging, and it is usually large.

2. Your demo questions had answers

Every question in the demo was answerable from the corpus, because you wrote them from the corpus.

A meaningful fraction of real traffic asks about things that simply aren’t there: features that don’t exist, policies from a document nobody uploaded, questions about last week when the corpus was indexed last quarter. A retrieval system always returns its top k — that’s what top-k means — so the model receives the k least-irrelevant chunks in the corpus and, unless instructed otherwise, writes a plausible answer from them.

This produces the highest-severity failure mode you have: a confident answer, with citations, to a question your corpus can’t answer. The citations make it worse, because they look like evidence.

How to measure it: put unanswerable questions in your eval set. Ten to twenty percent is a reasonable starting proportion, adjusted to what your logs actually show. The metric is abstention rate — how often the system says it doesn’t know when it shouldn’t know. Most systems that have never been tested this way score close to zero.

How to fix it: a relevance floor on retrieval scores, an explicit instruction in the prompt that “I don’t have information about that” is a valid and preferred answer when the context doesn’t cover the question, and — most effectively — a check that the retrieved context actually addresses the question before generating at all.

3. Your corpus was clean, small, and recent

Demos run on twenty hand-picked documents. Production runs on nine thousand, including a scanned PDF from 2019 whose text layer is garbage, four versions of the same policy with no indication which is current, an HTML export with the navigation menu embedded in every chunk, and a spreadsheet that parsed into a wall of unlabelled numbers.

Scale changes retrieval difficulty by itself — more documents means more near-duplicates competing for the same top-k slots. Quality changes it more.

How to measure it: sample fifty chunks at random from your production index and read them. Not the documents — the chunks, exactly as they’ll be handed to the model. Count how many are comprehensible in isolation. This takes half an hour and it is the highest-yield diagnostic in this entire post; people are consistently shocked by what’s in there.

Also measure: what fraction of retrieved chunks come from documents superseded by a newer version. Stale-but-relevant is a nasty failure because retrieval scores it highly and it’s wrong.

4. You asked one question at a time

Demo questions are self-contained. Production traffic includes follow-ups — “what about for enterprise customers?” — which are meaningless as standalone retrieval queries. Embedding “what about for enterprise customers” retrieves approximately nothing useful.

If your system passes the raw user turn to the retriever, multi-turn conversations degrade sharply after the first message, and it won’t show up in any eval set built from single questions.

How to measure it: build a portion of your eval set as conversations, not queries. Score retrieval on turns two and three specifically. How to fix it: query rewriting — a cheap model call that turns the conversation so far into a standalone search query — is the standard approach and it’s usually the single biggest improvement available to a chat-shaped RAG system.

5. You were a forgiving reader

You knew what the right answer looked like, so you read the output and confirmed it. Users read the output to learn the answer, which means they can’t detect a subtly wrong one — and a subtly wrong answer is worse than an obviously wrong one, because it gets acted on.

There’s also a volume effect. Fifty demo queries at a 90% success rate produce five failures, which feels like a good system. Fifty thousand production queries at the same rate produce five thousand failures, which feels like a broken one. The rate didn’t change; the absolute count did, and people respond to counts.

How to measure it: decide the acceptable failure rate before launch, in the open, with the people who’ll be looking at the failures. “One in twenty answers will be wrong” is a very different conversation held in advance than held in response to a screenshot.

The pre-launch checklist

Everything above, condensed:

1. Fifty questions written by someone who hasn’t read the corpus. Score retrieval on them separately.
2. Ten to twenty percent unanswerable questions. Measure abstention.
3. Read fifty random chunks from the production index. Count how many make sense alone.
4. Multi-turn conversations in the eval set, scored on turn two.
5. An agreed acceptable failure rate, stated before launch.

Every item produces a number you can put in a document. None of them require infrastructure you don’t already have.

Where to go next: building a retrieval eval set for the construction details, and the five places a RAG pipeline breaks for turning a specific bad answer into a specific fix.