What a RAG evaluation harness actually needs to measure

A retrieval-augmented generation system has two stages that fail independently, and almost every evaluation setup collapses them into one number. The number goes up, the system does not get better, and nobody can say why.

The fix is unglamorous: measure retrieval and generation separately, against a small set of questions a human wrote on purpose.

The two failures look identical from the outside

A user asks a question and gets a wrong answer. There are two very different causes:

1. The right passage was never retrieved. The model answered from its own weights, or from a passage that was topically close and factually irrelevant. No prompt change fixes this.
2. The right passage was retrieved and the model ignored it, contradicted it, or blended it with something else.

These need opposite responses. The first is a chunking, embedding or query-rewriting problem. The second is a prompting, context-ordering or model-choice problem. An end-to-end score that mixes them tells you the system is at 0.62 and leaves you guessing which lever to pull — so teams pull both, and each change moves the number a little for reasons nobody can attribute.

Stage one: is the answer in the context at all?

The retrieval metric that matters is not cosine similarity. It is: for this question, was at least one passage containing the answer inside the top k results?

That requires knowing which passage contains the answer, which means labelling. There is no way around it, and the good news is that the set can be small.

def recall_at_k(examples, retrieve, k: int = 5) -> float:
    """Fraction of questions whose gold passage appears in the top k."""
    hits = 0
    for ex in examples:
        retrieved = [doc.id for doc in retrieve(ex.question, k=k)]
        if any(gold in retrieved for gold in ex.gold_passage_ids):
            hits += 1
    return hits / len(examples)

Track it at more than one k. Recall@20 that is much higher than recall@5 means the retriever finds the passage and ranks it badly — a reranking problem, not an embedding problem, and a much cheaper fix.

Also track the negative case explicitly: questions the corpus genuinely cannot answer. A retriever returns its top k regardless, so those questions always come back with confident-looking irrelevant context, and the generation stage is what has to refuse. If your evaluation set has no unanswerable questions, you have not measured the behaviour users complain about most.

Stage two: given this context, is the answer supported?

With retrieval isolated, generation can be measured on a fair test: feed the gold passage directly, bypassing the retriever, and ask whether the answer is correct and grounded.

Three things are worth separating:

Groundedness is the one that catches the failure people actually fear. A model can produce a correct answer that is not supported by the retrieved context — correct because it memorised it — and that system will be spectacularly wrong the first time it is pointed at a corpus it has not memorised.

An LLM judge is a reasonable way to score these, with two caveats worth stating plainly. It must see the context and be asked a narrow, checkable question ("is every claim in this answer supported by this passage — yes or no, with the sentence that supports it"), not a vague one ("rate this answer 1-5"). And it needs its own calibration: score fifty examples by hand, compare, and know the judge's agreement rate before trusting the other nine hundred.

The evaluation set is the deliverable

The temptation is to generate the evaluation set with a model. It produces a thousand questions in an hour, and it is close to worthless, because a model generating questions from a passage produces questions that are answerable from that passage in the phrasing of that passage. Retrieval on such a set looks excellent and predicts nothing.

A better set is small and deliberately built:

• 30–50 questions is enough to start. Below that, one example moves the score by 2%; above a few hundred, the marginal question stops telling you anything new.
• Take them from real user questions if any exist — a support inbox, a search log, the questions people ask in chat.
• Include the awkward categories on purpose: multi-hop questions needing two passages, questions whose answer changed between document versions, questions using the user's vocabulary rather than the document's, and unanswerable questions.
• Store the gold passage IDs, not just the gold answer. Without them, stage one cannot be measured at all.

Keep it in version control next to the code. It is the single artefact that makes every future change to the system decidable, and it is worth more than the pipeline it tests.

Run it on every change

The harness earns its keep when it runs automatically — on every change to chunk size, embedding model, prompt, reranker or generation model — and reports the metrics separately:

recall@5      0.78  (was 0.71)   +0.07
recall@20     0.91  (was 0.90)   +0.01
groundedness  0.84  (was 0.86)   -0.02
refusal       0.62  (was 0.61)   +0.01

That output makes the tradeoff visible: retrieval improved, groundedness slipped slightly. Whether that is a good trade is a judgement call — but it is now a judgement call with evidence, which a single blended score never allows.