Rigorous evaluation of RAG systems
Identify errors at each stage of retrieval-augmented generation.
Index evidence, not just text
A RAG system selects candidate passages and supplies them to a model to draft an answer. Document identity, edition, date and rights should remain attached to indexed fragments; otherwise citations lose context.
Separate retrieval recall from faithfulness
A system can retrieve relevant passages yet write an unfaithful answer, or generate plausible prose from irrelevant evidence. Evaluate retrieval, reranking, faithfulness, citation accuracy and abstention when evidence is missing separately.
Build an evaluation set
Prepare positive, negative, ambiguous and conflicting questions with reference evidence. Separate training, tuning and evaluation examples to reduce overfitting to test cases.
Measure failures
A fabricated citation is a different failure from irrelevant retrieval. Report results by error class, with auditable examples and dates for corpus and model versions.
Failure analysis across pipeline stages
A RAG system can retrieve valid evidence and still distort a nuance, or retrieve unrelated text and write persuasive prose. Evaluate corpus coverage, retrieval, ranking, answer faithfulness and citation accuracy separately. Include no-evidence cases to test meaningful abstention rather than fabrication.
Exercise: auditable evaluation matrix
Build a table linking each question to expected passages, possible answers and human-assessed confidence. For each output, record actual retrieved passages and unsupported claims. Compare chunking or reranking variants with a fixed test set and publish persistent errors.
Core steps
- 01Prepare reference questions
- 02Measure retrieval separately
- 03Evaluate answer faithfulness
- 04Record corpus and model versions
Three questions to ask
Is retrieved evidence actually relevant?
Does each citation support its claim?
Does the system abstain when evidence is absent?
Read the source documents
Versions, reuse terms and document status can change. Check them on the original sites.