Why multi-hop questions break classic RAG (and what to do about it)
Top-k retrieval is a single hop by construction. When an answer needs three, the bottleneck stops being the model and becomes the shape of your corpus.
Ask a retrieval system this: "which supplier is affected by the export rule that changed after the incident in the Rotterdam warehouse?" Then watch it answer confidently about the wrong supplier.
The model is not the problem. The documents are not missing. The question needs three hops, and the retriever only knows how to take one.
Top-k retrieval is a single hop, by construction
Classic RAG is a straight line: embed the question, fetch the k nearest chunks, paste them into the prompt, generate. That is excellent when the answer lives in one place and the question already contains the words that place uses.
Multi-hop questions break that assumption. The chunk that names the Rotterdam incident does not mention export rules. The chunk with the rule change cites an incident number, not a warehouse. The supplier record mentions neither. Each intermediate document is irrelevant to the question as asked and becomes relevant only once you have read the previous one. A single similarity search cannot rank documents by a relevance that does not exist yet.
The number that made us write it down
On a benchmark where every question requires at least three chained hops, the same 12B local model scored 0 out of 11 as a classic top-k RAG reader and 11 out of 11 as a forest navigator, at 0.58× the tokens per correct answer of an iterative-RAG baseline, all on a single RTX 3090.
Eleven questions is a small benchmark and we say so plainly: the runner, the corpus and the baselines are committed to the repository so you can reproduce the numbers, disagree with them, or run them against your own data. The paper is a preprint.
The usual patches, and where each one stops
Raise k. More context means more distractors. Accuracy on middle-of- context facts degrades, and you pay for every token either way.
Rewrite the query (HyDE, expansions). A better first hop is still one hop.
Iterate (ReAct-style loops). This genuinely helps, and it is the honest baseline to beat. But every iteration runs a fresh similarity search over the same undifferentiated chunk soup. Nothing the loop discovers is written down, so the next question starts from zero.
Build a graph. The right instinct. It usually turns into an extraction pipeline that emits triples, which then get flattened back into text for the prompt: the structure is used once, at build time, and thrown away at query time.
What actually fixes it
- Give the corpus a shape a model can walk. Typed nodes, each carrying a short curated summary that says what the node is for. Parent/child branches. Explicit edges between related nodes.
- Turn retrieval into a sequence of cheap decisions. Land somewhere with search, look around, move along an edge, read one node in full. Each step is token-budgeted and each truncation is explicit, so the agent always knows what it has not seen.
- Let successful traversals leave a trace. When a hunt works, mint the shortcut it discovered and mark the trail. The corpus improves with use instead of staying frozen at ingest time.
- Measure hops, not just answers. If you cannot see how many steps an answer took, you cannot tell a lucky guess from a grounded one.
The honest version
If your questions are single-hop lookups (policy pages, product FAQs, support macros), classic RAG is simpler and cheaper, and you should keep it. The moment answers require joining facts that live in different documents, the bottleneck stops being the model and starts being the shape of your corpus. That is the part you control.