Retrieval Evaluation Sets: Testing Whether Search Actually Helps

What a retrieval evaluation set contains

A query alone is not a complete test case. Evaluators also need to understand the underlying need and what would count as useful evidence.

A practical test case can record:

• The information need: what the person is trying to learn or accomplish.
• The query: the wording submitted to the retrieval system.
• Relevant context: the user’s role, applicable product or location, time period, and access permissions where these affect the answer.
• Expected evidence: documents, passages, or facts that would support a useful response.
• Relevance judgments: an assessment of how well candidate results meet the need.
• Known limitations: missing sources, unresolved disagreement, or uncertainty about the expected result.

The evaluation run should also identify the collection being searched and the configuration being tested. A result is difficult to interpret if the underlying documents, filters, or ranking settings changed without being recorded.

This structure extends the broader practice of evaluating retrieval quality: it turns general questions about usefulness into cases that can be inspected and repeated.

Choosing representative information needs

An evaluation set should reflect the work the search system is intended to support. There is no universal collection of queries that establishes retrieval quality for every setting.

For a public website, useful cases might come from search logs, reader questions, support requests, and common navigation problems. For an internal knowledge system, they might come from recurring staff tasks, documented incidents, and questions that require finding a specific policy or procedure. Use real queries only where their collection and use are permitted, and remove sensitive details that are unnecessary for testing.

Include ordinary questions and consequential exceptions

Frequently asked questions matter, but frequency is not the only consideration. An uncommon query may deserve explicit coverage if a poor result could lead to a consequential mistake.

A balanced set may include:

• Exact names, identifiers, or phrases.
• Plain-language descriptions that do not use the source’s terminology.
• Ambiguous questions that need clarification.
• Questions requiring evidence from more than one source.
• Questions whose answers depend on a date, version, or jurisdiction.
• Questions for which the collection contains no adequate answer.
• Cases where relevant information exists but is outside the requesting user’s permissions.

These categories expose different retrieval conditions. Success with exact product names does not establish success with descriptions of a problem whose name the reader does not know.

Avoid writing every query from the source document

If evaluators read a document and then write a query using its heading and distinctive phrases, they may create an easier task than actual users face. Some source-derived queries are useful, but they should not stand in for the full range of information needs.

Include alternative expressions of the same need. The relationship between a person’s question and its search wording is explored further in query reformulation.

Judging evidence, not just matching words

Relevance judgments describe how well retrieved material serves a particular information need. They should assess usefulness in context, not simply whether a result mentions the topic.

A straightforward judgment scale might distinguish:

• Directly useful: supplies evidence needed to answer the question under the stated conditions.
• Partially useful: contributes necessary context or part of the answer but is insufficient alone.
• Related but insufficient: discusses the subject without supporting the requested answer.
• Not relevant: does not help resolve the need.

The definitions matter more than the labels. Write down how evaluators should handle outdated policies, near-duplicate pages, incomplete passages, and sources that apply to a different product or location.

Judge the unit the system actually returns

A relevant document does not make every passage within it relevant. If a system returns chunks, assess whether the returned chunk contains enough evidence to be useful. A paragraph referring to “the exceptions above” may be difficult to interpret when retrieved without those exceptions.

This is where chunking for retrieval becomes part of evaluation rather than merely an indexing decision.

Preserve uncertainty and disagreement

For important or difficult cases, have another qualified reviewer assess the evidence. Disagreement may reveal an unclear question, conflicting sources, or a judgment rule that needs refinement.

Do not turn an unresolved judgment into a definitive label simply because a scoring process requires one. Record uncertainty, resolve it where possible, and keep disputed cases distinguishable from settled ones.

Automated judges can assist with larger collections, but their agreement with human judgments should be checked. A fluent assessment is not, by itself, evidence that the relevance decision is correct.

An example: evaluating an internal knowledge search

Consider a hypothetical organization testing search across its current policies, archived policies, and support documentation.

One test case asks:

“Can I expense a hotel when my return flight is cancelled?”

The information need is not simply to find documents about hotels. The employee needs the applicable policy for an unplanned overnight stay, including any approval or documentation requirements.

The expected evidence might include:

• The current policy covering travel disruptions.
• A referenced section explaining approval requirements.
• Instructions identifying the documentation required for reimbursement.

A hotel-booking guide may share many query terms while failing to answer the question. An archived travel policy may answer it clearly but no longer apply. A current policy excerpt may be relevant yet incomplete if the retrieval system omits a linked exception.

A second case could ask what the policy allowed during an earlier year. The archived version might then become the appropriate evidence. This is a question of temporal relevance: usefulness depends on the period the question concerns, not automatically on which document is newest.

A third case might ask about a situation the policies do not address. The expected behavior should acknowledge the evidence gap and direct the user toward clarification—not treat a loosely related passage as a complete answer.

Together, these cases test distinctions that a single successful search would conceal.

Measuring retrieval results

Choose measures according to what the system needs to accomplish and how many results a person or downstream process can actually use. Evaluating the top five results may be more meaningful than evaluating the top fifty if only five passages reach the answer-generation stage.

Precision at a cutoff

The proportion of results within the chosen cutoff judged relevant. Precision at five asks how many of the first five results are useful under the evaluation’s relevance rules.

Recall at a cutoff

The proportion of all relevant items retrieved within that cutoff. Recall matters when missing evidence would leave the task incomplete, but it requires knowing—or estimating—which relevant items exist.

Reciprocal rank

A measure based on the position of the first relevant result. It is useful for tasks where finding one suitable result quickly is the main goal, but it says little about whether the system retrieves all necessary evidence.

Normalized discounted cumulative gain

Often abbreviated as nDCG, this measure accounts for graded relevance and result position. It can help assess whether more useful results appear earlier, provided the relevance grades are meaningful.

Account for incomplete judgments

In a large collection, evaluators may not know every relevant document. A common approach is to review a pool of candidate results gathered from several retrieval methods, then judge those candidates.

That creates a useful working reference, not necessarily an exhaustive inventory. An unjudged result is not automatically irrelevant. A new retrieval method may find good evidence that earlier methods never surfaced.

Report this limitation, especially when interpreting recall. A score measured against known relevant items should not be presented as proof that all relevant evidence was found.

Measure evidence coverage and appropriate restraint

For multi-part questions, count whether the retrieved material covers the required evidence—not just how many relevant passages it returns. Five near-duplicate passages may cover less of the need than two complementary ones.

For unanswerable cases, inspect whether the wider system recognizes insufficient support. For permission-sensitive cases, verify that unauthorized material is excluded. These conditions deserve explicit checks rather than being hidden inside a general relevance average.

Reading failures as patterns

An aggregate score summarizes performance, but individual cases explain what needs attention.

Review failures by the stage and condition involved:

• Coverage failure: the necessary evidence is absent from the searchable collection.
• Discovery failure: the evidence exists but does not enter the candidate results.
• Ranking failure: useful evidence is found but placed below less useful material.
• Passage failure: the right document is found, but the returned excerpt lacks essential context.
• Applicability failure: the evidence concerns the wrong date, version, location, or user context.
• Answer failure: sufficient evidence is retrieved, but the generated response misuses or omits it.

These distinctions prevent the wrong repair. Changing an embedding model will not supply a policy that was never indexed. Improving retrieval will not necessarily correct an answer generator that ignores qualifications.

In retrieval-augmented systems, evaluate retrieval separately from the final answer, then examine their relationship. Grounding AI-assisted answers depends on both obtaining suitable evidence and using it faithfully.

Comparing changes fairly

When testing a new retriever, filter, chunking method, or reranking stage, run the same evaluation cases against the baseline and the proposed configuration under comparable conditions.

Keep a record of the corpus snapshot, indexing settings, query transformations, result cutoff, and judgment version. Compare cases directly: which improved, which worsened, and which remained unchanged?

A higher average can conceal a serious regression in a small but important category. Review exact-identifier queries, historical questions, multi-source needs, and other meaningful groups separately.

For small evaluation sets, a change affecting one or two cases can move the average substantially. Report the number of cases and the observed changes rather than implying more certainty than the evidence supports.

Keep some cases out of routine tuning

If developers repeatedly adjust a system against the same examples, the system may become well suited to those examples without improving more generally.

Maintain a separate held-out set for periodic assessment. Keep closely related variants of the same information need together when separating tuning and test cases; otherwise, nearly identical questions can give a misleading impression of independent testing.

Repeatable checks of established cases support retrieval regression testing. They help reveal when an improvement in one area disrupts behavior that previously worked.

Maintaining the evaluation set

An evaluation set is a maintained reference, not a permanent statement of what every future result should be.

Sources change. New information needs emerge. A previously correct expected passage may become outdated, and a previously unanswerable question may gain a documented answer.

A practical maintenance routine is to:

1. Begin with a manageable collection of genuine information needs.
2. Document expected evidence, judgment rules, and known uncertainty.
3. Run a baseline and inspect results case by case.
4. Add failures and underrepresented conditions as they become visible.
5. Version changes to cases, judgments, and source collections.
6. Retain enough history to explain why results changed.

The provenance of the evaluation records matters too. Future reviewers should be able to tell who made a judgment, which sources informed it, and what conditions applied.

A useful retrieval evaluation set does not promise that search will always succeed. It makes success, failure, and uncertainty easier to examine across a meaningful range of needs. That provides a firmer basis for improvement than a handful of impressive examples—and a clearer account of where human review still matters.