Inside the 11-Question Benchmark
Eleven questions, each needing at least three chained hops. The same 12B model scores 0/11 as a top-k RAG reader and 11/11 as a forest navigator. Here is how the benchmark was built and how to rerun it.
4 min read
Blog
Notes from the forest
Engineering notes on navigation, ingestion, agent memory and honest benchmarking, written by the people who ran the experiments.
A Three-Hop Question, Worked Example
One question, three documents, traced twice: watch top-k retrieval dead-end on a three-hop question, then watch the same corpus answer it when an agent walks it node by node.
4 min read
Navigate, Don't Retrieve: What a Hunt Actually Looks Like
Three moves, one worked hunt through a small company's corpus, and why the same 12B model goes from 0/11 to 11/11 when it walks a forest instead of reading a top-k paste.
4 min read
Your agentic RAG loop answers slower and spends more per correct answer. We measured it.
Iterative RAG loops hide their cost in the wrong denominator. Measured per correct answer: 0.58x the tokens and 8.4 s p95 vs 17.5 s, same 12B model.
3 min read
Why multi-hop questions break classic RAG (and what to do about it)
Top-k retrieval is a single hop by construction. When an answer needs three, the bottleneck stops being the model and becomes the shape of your corpus.
4 min read
Want the long version?
The paper carries the full architecture, the benchmark tables and the findings that failed their criteria.