Evaluating RAG: RAGAS, ARES and Evidence-Centred Experiments
Published:
Begin with the failure you want to detect
A retrieved passage can be relevant but insufficient. An answer can be supported but incomplete. A correct answer can cite the wrong source. These cases need different labels, so define a schema before selecting metrics.
For each question, record the corpus snapshot, any acceptable answers, supporting evidence, the retrieved set, the selected context and the final claims. Keep unanswerable questions and conflicting-source cases in the test set rather than assuming every query has one clean answer.
| Layer | Example measurement | What it cannot establish alone |
|---|---|---|
| Retrieval | Supporting-evidence recall@k | Correct use by the reader |
| Context selection | Sufficient evidence retained | Final answer faithfulness |
| Generation | Correctness and requested coverage | Accurate source attribution |
| Attribution | Claim-level citation support | Completeness of the answer |
| Operation | Calls, tokens, latency, freshness | Linguistic quality |
What RAGAS contributes
The RAGAS paper develops automated evaluation of retrieval-augmented generation, including faithfulness, answer relevance and context quality. Its reference-free emphasis is useful when full human reference answers are expensive, but it does not make the evaluator infallible.
Do not assume every metric in every later software version has identical inputs or is reference-free. State the paper-style definition or the exact implemented metric, judge model, prompt and version used in the experiment. This chapter describes evaluation ideas rather than a moving library API.
A claim-based example
Suppose an answer contains five atomic claims, four supported by the supplied context and one unsupported. A simple support fraction is \(4/5=0.8\). That score changes if a sentence containing three claims is counted as one unit, or if the annotator disagrees about what follows from the source.
An answer repeating a supported trivial fact five times could score perfectly under that fraction while failing the request. Pair faithfulness with answer coverage and relevance. Also distinguish claims unsupported by the retrieved context from claims that are false in the world; those are overlapping but different categories.
What ARES adds
ARES uses synthetic training data for lightweight evaluation judges and human-labelled examples to support statistical estimation, including prediction-powered inference. It aims to estimate dimensions such as context relevance, answer faithfulness and answer relevance efficiently while accounting for judge errors.
A simplified correction intuition is
If the judge’s large-pool mean is 0.8 and a representative labelled sample reveals an average overestimate of 0.1, the corrected estimate is 0.7. This illustrates residual correction, not a complete confidence-interval implementation. Sampling, dependence and the chosen estimator determine valid uncertainty calculations.
A judge needs a test set too
Validate automated judgements on wrong-entity passages, negated claims, partial support, stale versions and plausible hallucinations. Judge agreement with humans can vary by domain and answer length. Long polished answers may receive favourable style-based assessments even when they contain unsupported details.
Use blinded comparisons and inspect disagreements. If model A generated both the answer and the evaluation data used to train the judge, correlated biases deserve special attention. A second model’s agreement is not automatically independent evidence.
Isolate retrieval from generation
Run oracle-evidence, shuffled-evidence and no-evidence conditions. A good answer with shuffled evidence may reveal reliance on model memory; a bad answer with oracle evidence reveals a reader or task-format bottleneck.
For retriever transfer, BEIR is a useful related benchmark family, but retrieval benchmark gains do not establish end-to-end RAG gains. RAGTruth supplies another relevant direction: human annotations of hallucination in RAG outputs can inform targeted faithfulness checks. Keep benchmark task definitions separate rather than merging their scores.
Report uncertainty and budgets
Pair systems on the same questions, resample at the independent unit and include training or generation variation where material. Match or report retrieval calls, context tokens and judge cost. Keep a representative human audit alongside automated scores, especially when selecting among many pipeline configurations.
Recap
- Measure evidence access, answer support and usefulness separately.
- RAGAS supplies automated dimensions; definitions and versions still matter.
- ARES uses judge learning and human-informed statistical estimation.
- Use controlled evidence ablations, paired comparisons and human audits.
References
- Es et al. Ragas: Automated Evaluation of Retrieval Augmented Generation. EACL System Demonstrations, 2024.
- Saad-Falcon et al. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. NAACL, 2024.
- Thakur et al. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks, 2021.
- Niu et al. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. ACL, 2024.