How we evaluate legal RAG systems (accuracy is the wrong metric)

By Dr. Musa Abdulkareem ·

Retrieval-augmented generation is the default architecture for legal document and case analysis, and for good reason. But the way most teams evaluate these systems — a "does the answer look right?" spot-check — is exactly the evaluation that lets a plausible, well-written, wrong answer through. In law, that's the only failure that matters.

A retrieval-augmented generation system for legal work looks, architecturally, like any other RAG system: chunk a corpus, embed it, retrieve the top-k chunks for a query, and have a language model synthesise an answer grounded in what was retrieved. The plumbing is the same whether you're building a support chatbot or a contract-review tool.

The evaluation is not the same, because the failure modes have different costs.

When a support chatbot hallucinates, a customer is mildly misinformed. When a legal analysis tool states that a clause is enforceable, cites a case that supports it, and the case doesn't actually say that — a lawyer relies on it, advises a client, and the error propagates into a real decision with real liability. The tool didn't need to be wrong often. It needed to be wrong once, convincingly, on a question that mattered.

So the evaluation question for legal RAG is not "is the answer usually right?" It's "when the system is wrong, is it wrong in a way a lawyer will catch — or in a way that looks exactly like being right?" Optimising for average accuracy actively works against you here, because the most dangerous outputs are the confident, fluent, well-cited, wrong ones, and those are a small fraction of a test set. Average accuracy rounds them away.

The four things we actually measure

We decompose evaluation into four axes, measured separately, because a single blended score hides the trade-offs that matter.

1. Retrieval quality — did we surface the right authority?

Before the model writes a word, retrieval has already decided whether a correct answer is even possible. If the governing statute or the on-point case isn't in the retrieved set, no amount of generation quality will save the answer — the model will either refuse (best case) or synthesise something plausible from adjacent-but-wrong material (worst case).

We measure retrieval independently of generation: for a set of questions with known relevant documents (identified by lawyers), what's the recall@k? Crucially, recall matters far more than precision here. Retrieving ten chunks where two are relevant is fine — the model can ignore the noise. Retrieving ten chunks where none are the controlling authority is a silent failure that generation will paper over. We track recall of the controlling authority specifically, not just topical relevance, because legal relevance is not the same as semantic similarity — the most textually similar case is often not the one that governs.

2. Faithfulness — is every claim grounded in retrieved text?

Given what was retrieved, does the generated answer make any claim that the retrieved text does not support? This is the anti-hallucination axis, and it's the one general-purpose eval usually skips.

We evaluate it claim-by-claim: decompose the answer into atomic assertions, and for each one, check whether it is entailed by a specific retrieved passage. An answer can be 95% grounded and still fail, because the 5% ungrounded claim is the one the model invented to make the answer feel complete. We treat faithfulness as close to pass/fail per answer, not as a percentage to average — one unsupported material claim fails the answer.

3. Citation accuracy — do the citations exist and say what's claimed?

This is where legal RAG systems most visibly and most dangerously fail, and it deserves its own axis. Three separate checks:

4. Refusal calibration — does it know what it doesn't know?

A legal tool that answers every question is broken. The correct response to "the corpus doesn't contain authority on this" is to say so — not to produce a fluent answer from whatever was nearest in embedding space. We build eval cases specifically for questions the corpus cannot answer (out-of-jurisdiction, out-of-date, genuinely novel) and measure whether the system refuses or confabulates. A system that scores well on answerable questions but confabulates on unanswerable ones is more dangerous than one that's mediocre across the board, because it has trained its users to trust it.

Building the evaluation set

None of this works without a good gold set, and building one is most of the effort.

Our eval sets are built with lawyers, not by engineers guessing at questions. They include: routine questions with clear answers (to measure baseline competence), edge cases where the law is unsettled (to measure whether the system represents uncertainty rather than manufacturing a false confidence), and deliberately adversarial cases — questions with a tempting-but-wrong answer, questions whose most semantically similar authority is not the controlling one, and questions the corpus can't answer. The adversarial subset is where systems separate, and it's the subset a "does this look right?" review never generates, because you don't think to test the cases you don't expect to fail.

We use LLM-as-judge to scale the faithfulness and citation-support checks — it's the only way to evaluate thousands of atomic claims affordably — but we calibrate the judge against human lawyer review on a sampled subset every cycle, and we never let the judge be the final word on the adversarial cases. A judge model shares failure modes with the model it's judging; trusting it unsupervised reintroduces exactly the risk you're trying to measure.

The compliance layer is part of the system, not a wrapper

One thing that doesn't show up in accuracy metrics at all: privileged and confidential data cannot leak between matters, into logs, or into a shared model's training data. For legal clients this is not a feature request; it's the precondition for the tool being usable at all. We design retrieval indices with matter-level access boundaries, keep privileged content out of any shared context, and treat data governance (PIPEDA, GDPR, privilege) as an architectural constraint from the first line of code — not something bolted on before launch. An analysis tool that is accurate but leaks privileged material across a wall is not a partially-good product. It's an unusable one.

The takeaway

Legal RAG is easy to demo and hard to trust, and the gap between the two is entirely in the evaluation. A system that answers ten sample questions impressively tells you almost nothing about whether it's safe, because the failure you care about is the confident wrong answer on the eleventh question — the one you didn't think to ask. Decompose the evaluation, measure retrieval and faithfulness and citations and refusal separately, build adversarial gold sets with the people who'll rely on the output, and treat a fabricated or misread citation as a hard failure rather than a rounding error. That's the work that turns a convincing demo into a tool a lawyer can actually put their name behind.

We build and evaluate lawyer-grade document and case-analysis systems for legal clients. If you're assessing a legal AI tool — or building one and want a rigorous evaluation harness around it — get in touch via the contact form on the main site.

Related capability: Legal & document intelligence

← All posts