I measured my RAG system before believing it

July 2026 · 12 min read

The eval suite for my RAG project was 10 queries over 5 documents, and it reported a perfect recall@5 of 1.00. Not because retrieval was perfect — because with five documents and top-5 retrieval, the system literally cannot miss. Every number that eval ever produced was a compliment, not a measurement. And I don't think my setup was unusual: most RAG pipelines I've seen are evaluated by a demo that looked right, a smoke test that saturated months ago, and everyone's memory of both.

I wanted to know what my retrieval upgrades were actually worth — chunking strategy, hybrid search, reranking — with numbers I could defend and rerun. So I built the measurement infrastructure first and made it open source: spanscore, an evaluation harness that scores RAG pipelines and agents from their OpenTelemetry traces. Then I ran the experiment: naive retrieval versus a properly engineered pipeline, under conditions anyone can reproduce. The improvement turned out to be real — and it looked nothing like I expected, or like any single number could describe.

Why eyeballing fails

Here's a concrete failure that never shows up when you spot-check a demo. One of my test queries asks: "What is the deductible on the premium tier of the home policy?" The correct answer — 150 EUR — sits in a coverage document, two lines below the other tiers. The baseline pipeline chunks documents into fixed 280-character windows, and the window boundary happened to fall between "the plus tier has a 300 EUR deductible" and the premium tier's line. The model answered:

"The deductible on the premium tier is not specified in the provided context. However, the plus tier has a 300 EUR deductible, and the basic tier has 500 EUR."

That answer looks great in a demo. It's grounded, it cites its source, it helpfully offers adjacent facts, and it's useless — the one number the user asked for was sliced off by a chunker. Retrieval "worked" (the right document came back), generation "worked" (nothing hallucinated), and the system still failed. You will not catch this class of bug by eyeballing, because every component looks locally correct. You catch it with a metric that checks whether the gold fact made it into the retrieved context — and that requires a gold set.

Building a gold set I could defend

A saturated eval set is worse than none, because it keeps emitting reassuring numbers. The fix is not more queries over the same tiny corpus — it's a corpus large and confusable enough that a retriever can actually be wrong, and gold cases precise enough to catch the chunk-boundary failure above. Eval sets sized for a smoke test quietly become the thing a team quotes; this one is sized to discriminate.

The benchmark is deliberately built to be confusable: 16 insurance policy documents, 60 cases, with near-collisions everywhere — four different deductibles (150, 300, 500, 1000 EUR) attached to different products, five different "within X days" periods, competing 5-percent discounts. Each case has the query, the relevant documents, a reference answer, and one or more gold facts: literal spans that must appear in the retrieved context for the case to be answerable. Two things I'd insist on in any gold set after this:

Grounding is checked by a script, not by discipline. A case only enters gold.jsonl if its gold facts are verbatim substrings of its source documents. Synthetic case generation drifts — the generator paraphrases, and suddenly your "sufficiency" metric measures paraphrase luck instead of chunking quality.
Disclose what the corpus is. Mine is a constructed benchmark written for this experiment, not a dump of the production corpus (which I couldn't publish anyway). What transfers is the methodology and the harness; your absolute numbers will differ.

The experiment: v1 vs v2

Both pipelines run end-to-end and emit OpenTelemetry-style spans — a retrieval span with the ranked documents, an LLM span with the generated answer and token usage. spanscore scores the traces, not the code: the harness never imports the pipeline.

v1 (baseline): fixed-size 280-character chunks + BM25 — the shape of most first RAG implementations.
v2: heading-aware chunking (one chunk per document section, title kept as context) + hybrid retrieval: BM25 and text-embedding-3-small fused with reciprocal rank fusion. Generation is gpt-4o-mini for both, temperature 0, answer-from-context-only with citations.

Every judge-backed metric below is a mean with a 95% bootstrap confidence interval — never a bare number. 60 cases each, identical for both systems:

metric	v1 naive+BM25	v2 heading+hybrid	Δ (95% CI, paired)
recall@1	0.850	0.875	+0.025 [+0.000, +0.067]
recall@3	0.933	1.000	+0.067 [+0.017, +0.133]
MRR	0.921	0.956	+0.035 [+0.008, +0.068]
context sufficiency	0.842	1.000	+0.158 [+0.075, +0.258]
faithfulness (judge)	0.919	0.983	+0.064 [+0.017, +0.131]
answer correctness (judge)	0.908	0.988	+0.079 [+0.021, +0.150]
citation coverage	0.850	0.931	+0.081 [+0.003, +0.167]

Three things I would not have said out loud before measuring:

The headline retrieval metric barely moved. Recall@1 improved by 2.5 points and the confidence interval touches zero — on this benchmark I cannot claim v2 finds the right document first any more often than v1 does. If I had only looked at recall, I'd have called the upgrade a wash.
The real win was never "retrieval accuracy". It was context sufficiency: +15.8 points, the largest and most significant delta on the board. Heading-aware chunking stopped slicing facts in half. That single change propagated downstream into +6.4 faithfulness and +7.9 answer correctness — the metrics a user actually feels.
No single number describes this upgrade. The honest summary is a vector: ranking barely changed, answerability changed a lot. The largest genuine relative improvement on the board is ~19% — on a metric I didn't have a name for before building the harness. If I had to compress this to one "retrieval improved by X%" soundbite, whichever metric I picked would misrepresent the other six.

Your LLM judge is a measurement instrument. Calibrate it.

Two of those metrics come from an LLM judge (gpt-4o-mini scoring answers on an anchored 1–5 rubric). Judges are everywhere now; judge calibration is almost nowhere. If a human wouldn't agree with the judge, the metric is theater. So I labelled all 60 baseline answers by hand against the same rubric the judge sees, and spanscore's calibrate command compared us:

$ spanscore calibrate results-v1.json labels-v1.csv
n=60 labelled cases, levels=[1, 2, 3, 4, 5]
weighted kappa (quadratic): 0.954
kendall tau-b: 0.890
exact agreement: 91.7%
mean bias (judge - human): -0.017

Quadratic-weighted Cohen's κ of 0.95 is high — the judge is usable for this rubric on this domain. But exact agreement is only 91.7%, and the confusion matrix showed the disagreements aren't random noise; they're two systematic habits:

The judge is harsh on honest refusals. Every answer of the form "the context doesn't contain this" got a 1 ("does not address the question") where the rubric says 2 ("unanswered"). An honest refusal and a hallucination scored identically — which is exactly the behaviour you don't want to optimize your pipeline against.
The judge is lenient about omissions. Answers that addressed the question but dropped a fact from the reference kept getting 5s where the rubric says 4. And in one rationale the judge confidently called two correct figures "incorrect information" while scoring — right verdict, wrong reasoning. It reads convincingly either way; that's the trap.

Neither pattern flips this experiment's conclusion — the biases are small and symmetric across both systems. But I only know that because I measured it, and now there's a number that says how much to trust the judge column. One design decision I'd defend hard: spanscore refuses to report κ on fewer than 30 labels. Agreement statistics on a handful of examples are noise wearing a lab coat; better no number than a false one.

Gating CI on evals without the gate crying wolf

A harness you run by hand is a harness you'll stop running, so the point of all this is a CI gate. The naive version — "fail if any metric dropped" — flaps constantly on judge variance and small samples, and a gate that flaps trains everyone to ignore it. spanscore's gate blocks only when a regression is statistically significant (a paired bootstrap over cases, so per-case correlation is preserved) and practically large (beyond a per-metric threshold you set in gate.toml). Noise warns; only real damage blocks. Running the gate "backwards" — v1 as a candidate against v2 as baseline, i.e. pretending someone shipped the downgrade — it blocks with exit code 1:

$ spanscore gate results-v1.json --baseline results-v2.json --config gate.toml
recall@3              block   -0.0667  [-0.1333, -0.0167]
context_sufficiency   block   -0.1583  [-0.2583, -0.0750]
answer_correctness    block   -0.0792  [-0.1500, -0.0208]
recall@1              pass    -0.0250  [-0.0667, +0.0000]   ← not significant
exit code 1

Note recall@1 correctly passing in both directions: the gate knows the difference between "worse" and "indistinguishable", which is precisely what keeps it credible. The traces and the judge cache are committed to the repo, so CI re-scores everything with no API key — a cold run of the whole experiment costs about $0.05; a warm run costs nothing.

What I'd cut with hindsight

The pipeline comparison was a day. The measurement infrastructure was the project. If I were starting again I'd skip the temptation to build evaluators for everything and start with exactly three: recall@k, context sufficiency, and one calibrated judge metric — that combination caught every real defect described here. I'd also write the gold-set grounding check first; it rejected several of my own hand-written cases for paraphrasing the corpus.

Everything here is reproducible from the repo: github.com/aawais-ai/spanscore — the harness (MIT, framework-agnostic, scores any OTel GenAI traces), the 60-case gold set, the committed traces, my hand labels, and the full interactive report this post's numbers come from. If you have a RAG pipeline you've been describing with an adjective — or a number you can no longer reproduce — pip install spanscore and spanscore init is the fastest way I know to find out what's actually true.

Next post: the same treatment for agent trajectories — scoring tool selection, argument correctness, and loop detection from OpenTelemetry spans, dogfooded on my sales-research-agent.