The problem is simple to state: how can a retrieval method be chosen before an answer is generated? The answer begins with the question’s shape. A request for one exact detail needs a different source-selection rule from a request that asks for an overview.
Retrieval is the step that selects source records for an answer. It happens before the language model writes. This order matters. If the wrong record enters the answer packet, clear writing cannot repair the missing evidence.
Our work on this problem uses a small permitted collection. It contains four records: three product descriptions named Cedar, Birch, and Alder, plus a return policy. The full collection contains only 299 characters. That small size makes the method easier to inspect. Every record has a preserved ID, version, path, and full text.
The collection does not get merged into one large prompt. Each record stays separate. This lets us ask a plain question: which records did the retrieval method select, and were those records enough for the question?
The question decides the retrieval need
A question can ask for one fact. “What is the product code for the two-meter USB-C cable?” is an exact-detail question. One product record should contain the needed answer.
A question can also use different words from the source. “How can I send it back?” may refer to a return policy without using the phrase “return policy.” This is an alternative-wording question. It still needs one source, but exact word matching may miss it.
Some questions need facts from separate records. A combined question might ask for a product’s physical details and the return rules. One product record cannot cover both parts. The selected set must include the product record and the policy record.
An overview asks for a wider set. “List every product and its physical details” requires all three product records. A short ranked list cannot cover that request if it contains only two records.
This is why task selection comes before method selection. The method is not chosen because one system seems generally clever. It is chosen because the question sets a known evidence need.
Four ways to select records
Word search uses an existing BM25 index. In plain terms, it looks for shared words and ranks records by their matches. It is useful for exact terms, such as a product code.
Semantic search compares 64-number representations of the question and each record. These representations are called embeddings. They can connect related wording even when the exact words differ. In this small collection, NumPy performs the comparison.
Hybrid fusion combines the word ranking and the semantic ranking. The system gives each record a position in each list, then joins those positions through reciprocal rank fusion. This can help when one method catches a useful signal that the other misses.
The fourth option is whole-source retrieval. It selects all four permitted records. It does not rank them for a narrow cutoff. It accepts a larger packet in exchange for broad coverage.
These methods use existing tools and saved results. No new model or library is added. The comparison restores locked dependencies, uses no cache, and keeps the files in an isolated lesson folder. These details may sound dull. They prevent a method comparison from changing because the software environment changed.
A small example shows the risk
Consider the exact question about AX17, the two-meter USB-C cable. Word search places Cedar first because Cedar contains AX17. Semantic search places Birch first because Birch contains the similar code AX170 and uses similar wording.
Fusion creates a harder case. Cedar and Birch receive ranks from the two lists. Their combined scores tie. The fixed source-ID rule places Birch first. The result is stable, but it is wrong for this exact question.
This example shows why a repeatable method is not automatically a good method. A deterministic error is still an error. It also shows why the task category matters. Word search is a better fit for the exact product code here.
The result changes for alternative wording. The semantic method recovers the return policy for “send it back.” Word search finds no result. That is a real retrieval benefit, not a claim that semantic search should control every question.
The combined case changes the cutoff again. If the selected list contains Cedar and Birch, it may still omit the return policy. Whole-source retrieval includes all four records and covers the separate evidence needs.
The overview case is clearer. No two-record list can contain Cedar, Birch, and Alder at the same time. The whole collection is required.
Freeze the choice before new questions
A fair comparison needs a fixed choice. The choice file records the question ID, its category, the selected method, the evidence IDs, the character count, and the original response identifier.
The category is assigned from the question’s form. It is not produced by the model. The chosen method is frozen before new questions or transfer questions are handled. Results from those later requests cannot influence the choice.
A SHA-256 fingerprint checks the choice file. Even a whitespace change creates a different fingerprint, so selection stops when the file differs. This check catches local changes. It does not protect against a simultaneous replacement of the file.
The answer packet contains the question and the selected source records. It does not contain the expected-answer rubric. That rubric is used later to measure coverage, not to guide the model. This separation matters because the answer process should not receive the test’s hidden target.
The comparison also avoids re-ranking. Saved rankings are checked against the exact source snapshot, and selections are recomputed from those rankings. Zero-score word matches are removed. Ties use an explicit source-ID rule. The measurement occurs after selection, when the selected IDs are compared with the required IDs.
Coverage counts how many required source IDs appear. A result might cover three of three required records. Character counts include source text and line breaks, but exclude labels, instructions, packet details, and model tokens. Coverage shows whether the needed records arrived. It does not prove that the answer used their facts correctly.
The practical choice
For this collection, the frozen choice is simple. Exact-detail questions use word search. Alternative-wording, combined, and overview questions use the whole permitted source.
Semantic embeddings remain available for comparison. They are useful for showing how related wording can recover a source that word search misses. Hybrid fusion also remains useful as an observed method. Neither becomes the primary choice here.
That decision is narrower than a claim about retrieval in general. It fits this collection, these question categories, and these output limits. A different source set could produce a different choice.
The main lesson is that retrieval should be selected from the evidence need. Exact details call for precise matching. Broad questions call for enough records to cover the request. Similar wording may call for semantic matching. When several records are needed, a narrow ranking cutoff can fail before the answer begins.
The project can now separate three things that are easy to confuse: a method’s ranking, a category’s evidence need, and an answer’s factual use of its sources. That separation makes errors visible. It also makes the system easier to teach, inspect, and revise without pretending that one retrieval method fits every question.
This is the kind of practical distinction The Source List is meant to hold together: one digital source worth knowing, one search tip, and one honest limitation.