Inside the 11-Question Benchmark
Eleven questions, each needing at least three chained hops. The same 12B model scores 0/11 as a top-k RAG reader and 11/11 as a forest navigator. Here is how the benchmark was built and how to rerun it.
The same 12B local model sat the same 11-question exam twice. As a classic top-k RAG reader it scored 0/11. Connected to a MonkeyLLM forest as a navigator it scored 11/11, and everything you need to check that claim is committed to the repo.
What strictly multi-hop means
A question is multi-hop when you need the answer to one sub-question before you can even phrase the next. We call it strictly multi-hop when at least three chained hops are required, and every one of the 11 questions in the benchmark clears that bar. The shape looks like this: to answer the final question you first need fact A, fact A gives you the terms to look up fact B, and only fact B tells you where the answer lives.
This is exactly the shape that breaks single-shot retrieval, because the keywords of the final hop do not co-occur with the keywords of the first. Top-k retrieval is one hop by construction, a point we unpack in why multi-hop breaks RAG.
How the questions were built
The corpus is the same set of 100 heterogeneous real-world documents used in our ingest measurement. Question construction was subtractive: draft a candidate, trace the minimal chain of evidence needed to answer it, and throw it out if any single document could settle it, or if a lucky keyword match could shortcut the chain. What survives is a set where the query terms for hop two only exist once hop one is answered.
That filter is brutal, which is one reason the set is small. Eleven questions is a small benchmark and we say so in the report. We would rather defend eleven questions we can trace hop by hop than a thousand we cannot audit.
The harness and the two runs
Both conditions use the same quantized 12B model on the same single RTX 3060 with 12 GB. Condition one is the default pipeline: chunk, index, retrieve top-k, hand the chunks to the model, grade the answer. Condition two hands the same model ten typed tools over MCP and lets it walk the forest (a forest is a graph of curated notes, each carrying a passport: title, summary, tags). It enters through search, reads labels, follows links, opens only what it needs, and every hop is token-budgeted. Navigate, don't retrieve.
0/11 to 11/11 is not a claim that the model got smarter. The weights never changed. The environment changed, and the paired scores measure exactly that gap and nothing else.
What is committed, including the failure
The corpus, the question sets and the harness are all committed, and every table in the report regenerates with one command. The paper is a preprint and we say so.
The report also keeps a criterion we failed: the trail-learning convergence criterion was not met, hops fell by roughly half the threshold we set. It is written down, not dropped, because a benchmark you can only pass is not a benchmark. The reasoning behind that choice has its own post.
You can also interrogate the corpus directly. The Studio's Data screen gives you read-only SQL over dataset nodes, which is the fastest way to convince yourself that no single row settles any of the eleven questions.
Regenerate it yourself
Skepticism is the correct response to a perfect score, so we made checking cheap. Clone the repo, follow the quickstart, and rerun every table with a single command. If your numbers disagree with ours, open an issue: that is what the harness is for. The rest of the story lives at monkeyllm.com.