← All articles
Model evaluation

Evaluating retrieval-augmented generation systems

Score retrieval and answer quality individually so failures can be traced to the right stage.

Check whether the evidence was retrieved

For each question, identify the passage or set of passages needed for a supported answer. Evaluate whether retrieval surfaces them. Some questions require combining multiple sources or distinguishing a current policy from a superseded one. Include those cases in the benchmark.

Check whether the answer uses the evidence

When the right passages are present, verify that each material claim is supported and that citations point to the relevant text. Include unanswerable questions and score whether the system acknowledges missing evidence. Keep retrieval misses visible even when the model gives a plausible answer from prior knowledge.

Separate a retrieval miss from a bad answer

Suppose a user asks whether a subscription can be canceled after renewal. Retrieval finds a general policy and an obsolete FAQ but misses a current exception. Record that the necessary passage was absent. If the passage was retrieved and the answer still missed its exception, record a generation failure. These diagnoses point to different fixes and should remain distinguishable in your report.

Build cases with well-defined evidence

Attach the relevant passage IDs and document versions to each question. Mark whether a single passage is enough or whether several are required. Include questions that cannot be answered from the corpus and questions where outdated sources conflict with current policy. For access-controlled corpora, evaluate with synthetic roles and verify that restricted documents are withheld before reaching the model.

Choose metrics that match the failure

Measure whether necessary evidence appears within the top retrieved results. For multi-source questions, count how often all required passages are present. Then evaluate whether the answer is correct, supported by its evidence, and correctly abstains when evidence is missing. Audit citations for their alignment with individual claims, because a link to a relevant document may still fail to support the sentence next to it.

Use an evidence comparison to plan the fix

Run the same questions with ordinary retrieval and with verified evidence supplied to the model. If the second run improves results, inspect indexing, chunk boundaries, or ranking. If both runs struggle, investigate context selection and response instructions. Record which sources were available and which settings were used so the experiment can be repeated after the next change.