Measuring Faithfulness, Not Just Answer Quality

Retrieval metrics tell you what reached the model. They say nothing about what the model did with it, and the most consequential failure in RAG happens entirely on the generation side: the model produces an answer that isn’t supported by the context it was given.

Sometimes that answer is even correct — pulled from the model’s parameters rather than your documents. That’s the hardest case to catch and the one that makes the system untrustworthy, because you cannot tell from the output which answers came from your corpus and which came from somewhere else.

Three different questions

Keep these apart. Nearly every muddled RAG evaluation muddles them.

Faithfulness / groundedness. Is every claim in the answer supported by the retrieved context? Purely a relationship between two texts. Doesn’t require knowing the truth.

Answer correctness. Is the answer true? Requires a reference answer or a knowledgeable human.

Answer quality. Is it complete, well-organised, appropriately hedged, at the right length?

An answer can be faithful but wrong — the retrieved document was outdated and the model reported it accurately. This is a retrieval or corpus problem, not a generation problem, and faithfulness scoring correctly declines to blame the model.

An answer can be correct but unfaithful — the model knew the answer independently and the context contributed nothing. It looks like a success and it is a warning: the system is producing output your corpus doesn’t back, and on the next question, where the parametric knowledge is stale or wrong, it will do the same thing with a worse result.

Faithfulness is the one that measures whether retrieval is doing its job. Measure it separately, always.

How to measure faithfulness

The tractable method is claim decomposition, in three steps.

1. Split the answer into atomic claims. Each a single verifiable assertion.

Answer: “Damaged items can be returned within 30 days of delivery, and return shipping is free for orders over £50.”

Claims:

• Damaged items can be returned within 30 days of delivery.
• Return shipping is free for orders over £50.

2. Check each claim against the retrieved context, independently. Three verdicts: supported, contradicted, not mentioned.

3. Score. Faithfulness = supported claims / total claims. Track contradicted separately — it’s a different severity from unsupported.

The reason to decompose rather than judge the whole answer at once: an answer with four supported claims and one invented one reads as fine holistically. Judged claim by claim it scores 0.8, and you can see exactly which sentence is the problem. Holistic judgements systematically miss the single fabricated detail embedded in a correct answer, which is the exact failure you’re hunting.

Doing it with a model

Both steps are LLM tasks. Keep the prompts narrow.

DECOMPOSE = """Split the answer into atomic factual claims.
One claim per line, each independently verifiable, no commentary.

Answer: {answer}"""

VERIFY = """Context:
{context}

Claim: {claim}

Is the claim supported by the context? Answer with exactly one word:
SUPPORTED, CONTRADICTED, or NOT_MENTIONED.
Judge only against the context. Do not use outside knowledge."""

Two things this prompt does deliberately: it forces a single-token verdict, which is far more reliable to parse and to compare than free text, and it instructs the judge to ignore its own knowledge — without which the judge marks true-but-ungrounded claims as supported, defeating the entire measurement.

Verify claims one at a time. Batching several claims into one call degrades accuracy and introduces order effects.

Where LLM judges fail

Necessary and imperfect. Both at once. The documented and widely reproduced weaknesses:

Verbosity bias. Longer answers tend to score better on quality dimensions regardless of content. Matters less for claim-level faithfulness, which is part of why claim decomposition is the more robust design.

Position bias. In pairwise comparisons, the option presented first is favoured. If you’re A/B testing two systems with a judge, run every comparison in both orders and average — this is not optional, and it’s cheap.

Self-preference. Judges tend to rate outputs from the same model family more highly. Use a different model as judge from the one generating, where you can.

Leniency on “not mentioned.” Judges under-report unsupported claims unless explicitly told that outside knowledge is off-limits — hence the instruction in the prompt above. Test this directly: feed it a claim you know is true and absent from the context, and confirm it returns NOT_MENTIONED.

Sensitivity to prompt wording. Rephrase the judge prompt and the scores move. Which means judge scores are comparable within one prompt version and not across versions. Version your judge prompt and record it with every result.

Calibrate against humans

Non-negotiable if the number is going in a decision.

Take 50–100 claim/context pairs, have a human label them with the same three verdicts, and compute agreement with the judge. What you want to know:

• Overall agreement rate. Below roughly 80% and the judge isn’t measuring your definition of faithfulness.
• Asymmetry. Does it over-mark SUPPORTED or under-mark it? A judge that’s lenient in a known direction is still usable — you’re reading a biased instrument with a known bias.
• Where it fails. Numeric claims, negations (“X is not covered”), and multi-part claims are the usual weak spots.

Recalibrate whenever you change the judge model or the prompt.

What to report

Metric	What it answers
Faithfulness (supported / total claims)	Is the answer grounded in what we retrieved
Contradiction rate	How often we assert the opposite of our own documents
Abstention rate on unanswerable questions	Do we say “I don’t know” when we should
Citation accuracy	Does each cited source actually contain the claim
Human agreement rate for the judge	How much any of the above is worth

That last row is the one people leave off, and it’s the one that makes the others meaningful.

Citation accuracy deserves a specific mention. If your system shows sources, a user’s trust is anchored on them — and citations that point at documents which don’t contain the claim are worse than no citations at all, because they manufacture confidence. It’s checkable with the same claim-verification machinery: verify each claim against its cited chunk specifically rather than the whole context.

When faithfulness is low

Diagnose in this order:

1. Is the context any good? Low faithfulness often means the model got nothing useful and improvised. Check retrieval metrics for those specific questions first — recall@k and what it hides.
2. Does the prompt permit it? “Answer using the context” invites blending. “Answer using only the context; if the context does not contain the answer, say so” is a different instruction and often the entire fix.
3. Is the context too long? Material in the middle of a long context gets used less reliably. Fewer, better chunks frequently beats more.
4. Is the question unanswerable? Then the correct output is a refusal, and low faithfulness is measuring an abstention failure, not a generation failure.

Which of those it is comes down to isolating the stage — the five places a RAG pipeline breaks.