Imagine asking an AI assistant to compare two ways of moving a website to a replacement platform: a phased migration, in which defined parts of the site move in stages, and a coordinated cutover, in which the in-scope site switches during a planned transition window.
The assistant retrieves several credible-looking pages, but every page explains phased migration. The material may help describe that approach. Does it support the requested comparison?
Not yet. Relevant results are not necessarily sufficient evidence.
Evaluating retrieval quality means checking whether the retrieved material supports the particular question, explanation, or decision at hand. That requires examining what each source establishes, how the sources fit together, and which consequential questions remain unanswered.
The migration scenario below is illustrative, not a report of an actual search or a recommendation about either approach. Its purpose is to show how a useful-looking source set can still leave an answer unsupported.
Define what the task needs
Before judging the results, establish what a satisfactory answer would have to explain. A definition, a comparison, and a recommendation have different evidence requirements.
For the migration example, assume the question concerns moving the same public-facing website to a replacement platform while retaining its domain. Here, “phased migration” means moving defined groups of pages or functions in stages. “Coordinated cutover” means switching the agreed scope together during a planned transition window. These are working definitions for this example; project teams may use the terms differently.
Even with those definitions, a recommendation needs more context. Can the platforms operate alongside each other? Do accounts, transactions, or content updates cross the proposed boundaries? What interruption is acceptable? Who can monitor and respond during the transition?
Those conditions determine which comparison questions matter. For this scenario, a reviewer might begin with four:
- Continuity: What must remain available throughout the transition?
- Dependencies: Which pages, functions, or systems must move together?
- Recovery: What would returning to the previous state require, including handling changes made after the transition began?
- Operational capacity: What testing, coordination, and monitoring can the team support?
This is not a universal migration checklist. It is a bounded set of information needs against which to evaluate the retrieved material. Broader website migration planning may introduce other requirements.
Separate missing project context from missing evidence
If the sources do not explain a relevant platform limitation, further retrieval may help. If nobody has specified how much interruption the organization can accept, another search cannot supply that decision.
Missing source evidence calls for investigation. Missing project context often calls for a question to the project owner. Confusing the two can produce more results without making the answer more useful.
Check relevance at the claim level
A source can be about website migration without supporting the statement an answer proposes to make. Matching terminology helps identify candidates; it does not establish that those candidates answer the question.
Consider three different claims:
- “A phased migration moves defined portions of a site in stages.”
- “This platform supports operating the old and new implementations together under these conditions.”
- “A phased migration is the better choice for this website.”
A descriptive overview might support the first claim. Applicable platform documentation might support the second. The third requires a comparison tied to the website’s constraints, with evidence for the tradeoffs that determine the recommendation.
A source supporting one claim does not automatically support the others.
For each consequential statement, ask:
- Does the material address this claim directly, or only the surrounding topic?
- Does it concern the relevant platform, configuration, or operating conditions?
- Is it describing a possibility, reporting an observed outcome, or making a recommendation?
- What qualifications limit how the statement can be used?
This is the practical distinction between finding related information and grounding an answer in identifiable sources. The connection must hold between the evidence and the actual claim, not merely between the source title and the article topic.
Examine coverage across the source set
Individual results may be relevant while the collection remains incomplete. Coverage concerns whether the set addresses the parts of the task needed for a defensible answer.
Return to the illustrative migration search. If all retrieved sources explain phased migration, the set may support a description of that approach. It has not yet established how a coordinated cutover would work under comparable conditions, or which differences matter for this site.
A small evidence map can make that limitation visible. The following map records only the hypothetical situation, not findings from reviewed documents.
| Comparison question | Current support | What remains unresolved |
|---|---|---|
| What does each approach involve? | The set contains explanations of phased migration. | An applicable explanation of coordinated cutover is still needed. |
| Can the site move in separate parts? | General explanations alone do not establish feasibility for this site. | Project dependencies and relevant platform capabilities need review. |
| How would recovery work? | The presence of migration-related pages does not establish recovery behavior. | Evidence is needed about restoration, changed data, and dependencies for each feasible approach. |
| Which approach fits the team? | No recommendation follows from one-sided descriptions. | Team capacity and the operational requirements of each approach remain necessary. |
The map does not require equal source counts. One directly applicable technical document may contribute more than several broad summaries. Nor does a comparison require equal endorsement: an option may prove unsuitable once a decisive constraint is established.
The purpose is to avoid mistaking an unexamined alternative for a disproven one.
Absence from the retrieved set is not evidence of absence everywhere. A limited search may have missed useful material because of its wording, source selection, access limits, or the way documents were indexed.
Assess source quality and independence
Relevance, credibility, and sufficiency answer different questions:
- Relevance: Does this material address the information need?
- Credibility: Is there a sound basis for relying on it for this claim?
- Sufficiency: Does the available evidence justify the proposed conclusion?
A reliable source can be relevant to one part of the question without making the whole answer sufficient.
Read origin, purpose, date, and scope together
Vendor documentation may establish supported product behavior, but it may not offer an independent comparison of migration strategies. A project account may describe what happened in one organization without showing that the same outcome should be expected elsewhere.
Currency also depends on the claim. Older material may explain a durable principle accurately. A recently updated page may still concern a different product version or an unsuitable configuration.
Look for an identifiable origin, a clear basis for the statement, relevant conditions, and enough context to inspect its limits. A polished website or high search position does not substitute for those checks.
Check whether multiple results provide independent support
Several pages may repeat one documentation statement, summarize the same case study, or cite the same original report. They are multiple results, but not necessarily multiple independent confirmations.
Deduplication helps identify repeated information. Evaluating independence goes further: it asks where the evidence originated. Different wording does not prove separate support, just as similar wording does not by itself prove a claim false.
Repetition can still help explain a concept. It should not be counted as additional corroboration without examining its origin, and it may crowd out material that answers a different part of the question.
Check what passage selection left out
A retrieved passage is a selected portion of a source. It may omit the definition, condition, exception, or date needed to interpret it.
For example, an extract might describe a transition as reversible while leaving out a nearby qualification about data written after the switch. Without that condition, an answer could turn a limited recovery statement into a much broader assurance.
When a consequential claim depends on an excerpt, inspect the surrounding material. Check whether a heading limits its scope, a preceding paragraph states an assumption, or a table note supplies an exception.
Also distinguish where the omission may have occurred:
- The full source may not contain the needed information.
- The retrieval step may have returned a passage that excludes it.
- The information may have been retrieved but omitted during later context selection.
These are different problems. The first may require another source; the second may require a broader passage; the third may require revisiting what was retained for the answer. Chunking documents without breaking meaning and context prioritization both affect what the final reviewer can see.
Choose the next action from the gap
Once a gap is visible, the next step should address it specifically. “Find more sources” is less useful than naming what the answer still needs.
- An alternative is undocumented: Seek applicable material about that approach.
- A project condition is unknown: Ask the responsible person, inspect project records, or state the unresolved condition.
- A claim exceeds its source: Narrow the statement or find evidence that supports the broader claim.
- A passage is ambiguous: Read its original context before using it.
- The results repeat one origin: Look for independent evidence if corroboration matters to the conclusion.
- Evidence remains limited: Explain what can be answered and what cannot yet be decided.
For the migration example, the immediate need is not another general introduction to phased migration. It is support for the missing side of the comparison, together with the site conditions needed to assess applicability.
Do not fill an empty evidence-map cell with a plausible benefit or drawback simply to finish the comparison. Reasoned inference can be useful, but it should remain distinguishable from directly supported statements. That separation is central to reporting fact, inference, and uncertainty accurately.
Stop when the remaining work no longer serves the answer
Evaluation does not require unlimited searching. A useful stopping point may be a supported narrow explanation, a conditional comparison with explicit limits, or a clear statement that a recommendation must wait for project information.
For a consequential decision, unresolved gaps may warrant further investigation. For a simple definition, the same breadth may be unnecessary. The stopping decision should reflect the task and the consequences of being wrong, not an arbitrary source-count target.
Understand what this review can establish
Reviewing a source set can show whether the material currently available supports a particular answer. It does not, by itself, measure the quality of an entire retrieval system.
System-level evaluation requires a broader design: representative queries, defined information needs, consistent relevance judgments, and an appropriate comparison basis. A careful review of one migration question can reveal a useful failure case, but it cannot establish performance across unrelated tasks.
For everyday retrieval-assisted work, the practical method can remain simple:
- Identify what the question requires.
- Inspect what the evidence actually supports.
- Record consequential gaps and limitations.
- Choose a next action that follows from those gaps.
No numerical score is needed to recognize that a two-option comparison currently contains evidence about only one option.
Leave a traceable conclusion
The final answer should preserve the outcome of the evidence review rather than hide it beneath fluent prose.
For the hypothetical migration example, an accurate status report could read:
Illustrative review status: The current material explains phased migration, but does not yet support a comparison with a coordinated cutover. Applicable evidence about the cutover approach is still needed. The site’s dependency boundaries, recovery requirements, acceptable interruption, and available team capacity also need to be established before recommending either approach.
A bounded explanation of phased migration is possible now. A recommendation between the two approaches is not yet justified.
In a real review, attach the relevant source references to supported claims and record unresolved questions where a later reviewer can find them. Keep the distinction between what was documented, what was inferred, and what remains unknown.
Good retrieval is not simply a collection of pages that sound relevant. Its value depends on whether the selected evidence can carry the answer being built from it—and whether the answer stops where that support ends.