Measuring retrieval: recall@k, MRR, and a golden set you'll maintain

Parts 1 and 2 made a dozen decisions: chunk size, overlap strategy, breadcrumb context, fusion weights, RRF_K, candidate depth, rerank cutoff. Every one was justified with an argument. None was justified with evidence.

That is the normal state of a RAG system, and it is why RAG work feels like guessing. Someone reports a bad answer, you widen the chunk size, the bad answer goes away, and three other things quietly get worse.

This part builds the measurement. It is less code than either previous part and it is the part that makes the other two improvable.

Understand what you're measuring#

Retrieval either put the right chunk in the context or it didn't. That's a binary, per query, and it is entirely separable from whether the model then wrote a good answer.

Keeping those separate is the single most valuable thing this harness does. "The answer was wrong" has two completely different fixes depending on which stage failed, and without measurement you cannot tell them apart.

Three metrics cover it:

Recall@k — of all the chunks that could have answered this query, what fraction appeared in the top k? This is the ceiling on everything downstream. If the relevant chunk isn't in the context, no amount of prompt engineering recovers it.

MRR (Mean Reciprocal Rank) — 1/rank of the first relevant result, averaged. Sensitive to ordering in a way recall isn't. Recall@10 can't tell you whether the answer sat at position 1 or position 10; MRR can, and position matters because you truncate at some point.

nDCG@k — handles graded relevance ("this chunk fully answers it" versus "this chunk is related"). Use it once binary relevance stops capturing your corpus, not before.

Track recall@5, recall@20, and MRR@10. The gap between recall@5 and recall@20 is diagnostic: large gap means your retriever finds the right material but ranks it badly (fix the reranker), small gap at low absolute recall means it isn't finding the material at all (fix chunking or the query side).

Bootstrap a golden set from your own corpus#

You need queries with known-correct chunks. Hand-writing 100 is a day of work people never do, so bootstrap it: have a model read a chunk and write the question that chunk answers. That chunk is then a known-relevant result by construction.

golden.py

import json
from anthropic import Anthropic
 
client = Anthropic()
 
PROMPT = """You are building a retrieval test set from a documentation corpus.
 
Read the passage. Write 2 questions a real user would ask that this passage
answers well.
 
Rules:
- Questions must be answerable from THIS passage alone.
- Use the vocabulary a user would use, not the passage's phrasing. If the
  passage says "authentication token expiry", a user might ask "why do I keep
  getting logged out".
- One question should use an exact identifier from the passage if it has one
  (an error code, parameter name, or function name).
- Skip the passage entirely if it is boilerplate, navigation, or a stub.
 
Return JSON: an array of objects with keys "question" and "kind"
("paraphrase" or "exact"). Return [] to skip."""
 
def generate_cases(chunk: dict) -> list[dict]:
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=PROMPT,
        messages=[{"role": "user", "content": chunk["raw_content"]}],
    )
    text = "".join(b.text for b in response.content if b.type == "text")
    try:
        items = json.loads(text)
    except json.JSONDecodeError:
        return []
 
    return [
        {
            "query": item["question"],
            "kind": item["kind"],
            "relevant_chunk_ids": [chunk["id"]],
            "source_uri": chunk["source_uri"],
        }
        for item in items
    ]

Sample the chunks you generate from — 150 or so, stratified across document types and token counts, so your test set isn't dominated by whatever document happens to be longest.

Label the near-misses#

The generated set has a specific flaw: it marks exactly one chunk relevant, but other chunks may answer the query just as well. Those count as misses, and your recall number comes out artificially low.

Fix it with one pass: run retrieval, look at what came back that wasn't the seed chunk, and judge it.

label.py

JUDGE = """Does this passage contain information that answers the question?
 
Answer with one word:
YES  - contains a complete answer
PART - contains some of the answer but not all
NO   - does not help answer the question"""
 
def label_candidates(conn, case: dict, k: int = 20) -> dict:
    hits = search(conn, case["query"], top_n=k)
    known = set(case["relevant_chunk_ids"])
 
    for hit in hits:
        if hit["id"] in known:
            continue
        response = client.messages.create(
            model="claude-opus-5",
            max_tokens=8,
            output_config={"effort": "low"},
            system=JUDGE,
            messages=[{
                "role": "user",
                "content": f'<question>{case["query"]}</question>\n'
                           f'<passage>{hit["raw_content"]}</passage>',
            }],
        )
        verdict = "".join(b.text for b in response.content if b.type == "text").strip()
        if verdict.startswith("YES"):
            case["relevant_chunk_ids"].append(hit["id"])
 
    return case

This runs once per case. Save the result to disk and never regenerate it — the labels are the asset, and regenerating them makes your metrics incomparable across runs.

Spot-check 30 of them by hand. If the judge disagrees with you more than about 10% of the time, fix the rubric before you trust any number it produced. A judge you haven't validated is a random number generator with good manners.

Compute the metrics#

metrics.py

from statistics import mean
 
def recall_at_k(retrieved: list[int], relevant: set[int], k: int) -> float:
    if not relevant:
        return 0.0
    return len(set(retrieved[:k]) & relevant) / len(relevant)
 
def reciprocal_rank(retrieved: list[int], relevant: set[int], k: int = 10) -> float:
    for i, chunk_id in enumerate(retrieved[:k], start=1):
        if chunk_id in relevant:
            return 1.0 / i
    return 0.0
 
def evaluate(conn, cases: list[dict], retriever) -> dict:
    per_case = []
    for case in cases:
        retrieved = [h["id"] for h in retriever(conn, case["query"], top_n=20)]
        relevant = set(case["relevant_chunk_ids"])
        per_case.append({
            "query": case["query"],
            "kind": case["kind"],
            "r@5": recall_at_k(retrieved, relevant, 5),
            "r@20": recall_at_k(retrieved, relevant, 20),
            "mrr": reciprocal_rank(retrieved, relevant),
        })
 
    return {
        "n": len(per_case),
        "recall@5": mean(c["r@5"] for c in per_case),
        "recall@20": mean(c["r@20"] for c in per_case),
        "mrr@10": mean(c["mrr"] for c in per_case),
        # Segment: exact-identifier queries are where vector-only search fails.
        "recall@5_exact": mean(c["r@5"] for c in per_case if c["kind"] == "exact"),
        "recall@5_para": mean(c["r@5"] for c in per_case if c["kind"] == "paraphrase"),
        "cases": per_case,
    }

That segmentation on the last two lines is the highest-value line in the file. An aggregate recall of 0.81 tells you almost nothing. Splitting it into 0.94 on paraphrase queries and 0.52 on exact-identifier queries tells you precisely where to spend the afternoon.

Run the comparison you've been arguing about#

Now every decision from parts 1 and 2 is testable. Because the schema kept strategy as a column, two chunkings can coexist and be measured against the same golden set.

compare.py

from functools import partial
 
VARIANTS = {
    "vector only":     lambda c, q, top_n: hydrate(c, [h.chunk_id for h in vector_search(c, q, k=top_n)]),
    "keyword only":    lambda c, q, top_n: hydrate(c, [h.chunk_id for h in keyword_search(c, q, k=top_n)]),
    "hybrid":          partial(search, rerank_enabled=False),
    "hybrid + rerank": search,
}
 
for name, retriever in VARIANTS.items():
    result = evaluate(conn, cases, retriever)
    print(f"{name:18} r@5={result['recall@5']:.3f}  r@20={result['recall@20']:.3f}  "
          f"mrr={result['mrr@10']:.3f}  exact={result['recall@5_exact']:.3f}")

Typical shape of these results, and how to read it:

Variant	recall@5	recall@20	mrr@10	exact
vector only	0.71	0.88	0.58	0.44
keyword only	0.63	0.79	0.51	0.91
hybrid	0.84	0.94	0.69	0.88
hybrid + rerank	0.84	0.94	0.81	0.88

Three things to notice, because they generalize.

Vector-only collapses on exact-identifier queries — 0.44 against keyword's 0.91. That single column is the entire argument for hybrid retrieval, and it is invisible in the aggregate.

Reranking does not change recall at all. It cannot: it only reorders what fusion already found. It moves MRR from 0.69 to 0.81, which is exactly what a reranker is for. If you ever see a reranker "improve recall," your harness is wrong.

The gap between recall@5 (0.84) and recall@20 (0.94) is where the remaining headroom lives. Ten percent of relevant chunks are being found but ranked between 6 and 20. That's a ranking problem, not a retrieval problem — more reranker, not more chunking.

Separate retrieval failure from generation failure#

Last piece. When an answer is wrong, you now have the machinery to say why in one query:

triage.py

def triage(conn, query: str, expected_chunk_ids: set[int]) -> str:
    retrieved = [h["id"] for h in search(conn, query, top_n=8)]
 
    if not (set(retrieved) & expected_chunk_ids):
        return "RETRIEVAL: relevant chunk never reached the context."
 
    return "GENERATION: chunk was in context; the model failed to use it."

Two words, two entirely different workstreams. RETRIEVAL sends you to chunking, fusion weights, and query handling. GENERATION sends you to the prompt, the context ordering, and the effort level — and no amount of retrieval tuning will help.

Most teams without this distinction spend their time on the wrong one. The symptom is identical; the fix is not.

Running it as a habit#

Wire it into CI on a schedule rather than per-commit — it costs real API calls, and retrieval quality does not change on every push. A nightly run against the golden set, with the numbers written somewhere you'll see them, catches regressions from corpus drift, dependency upgrades, and model changes.

Two rules keep the set honest over time:

Every production failure becomes a case. Someone reports a bad answer, you triage it, and the query goes in the file — labeled, permanently. This is how the golden set stops being synthetic and starts being a record of how your system actually breaks.

Never regenerate labels. Add to the set; don't rebuild it. The moment labels change, you lose the ability to compare against last month, and comparison across time is most of the value.

What the series bought you#

Structure-aware chunking with the context each chunk needs to be findable. Keyword and vector retrieval fused with a method that survives model swaps. Filtering that's a WHERE clause. And now, numbers.

The numbers are the part that compounds. Everything in parts 1 and 2 was a reasonable-sounding default. Half of them are probably wrong for your corpus — and now you can find out which half, one experiment at a time, instead of arguing about it.