Same Forest, a Fifth of the Model: 11/11 at 2B

Every number we have published so far came out of a 12B model. That was the point of the benchmark: a mid-range card, a quantized 12B, and a forest that made it score 11/11 where top-k RAG scored 0/11. This week we asked the obvious next question. If the environment is doing the work, how small can the model get before the score moves?

The swap

We took MiniCPM5-2B, an open 2.5B-parameter model from OpenBMB, quantized to Q8 and served by Ollama, and pointed the same navigator harness at the same forest: 153 nodes, two SQLite datasets, the same three question sets the paper reports. Nothing about the forest changed. Entry search ran BM25-only, with no embedder and no vector index, which is less retrieval help than the 12B rows had. One run per set, on 2026-09-11.

The scores:

• Mixed set, 18 questions: 18/18. The 12B scored 18/18.
• Strictly multi-hop, 11 questions, each needing at least three chained hops: 11/11. The 12B scored 11/11.
• Fork-tier, 8 questions with two to four parallel sub-chains: 7/8. The solo run in the paper scored 7/8 too.

Citation precision was 1.00 on the first two sets: every node the 2B cited was one it had actually opened, and it held the answer.

The model is a fifth the size and used less than half the memory: about 3.3 GB of VRAM against 7.6 GB for the quantized 12B. The forest is identical. The score is identical.

What it cost

Smaller models look around more. Median observation tokens per question went from 1,433 to 1,583 on the strictly multi-hop set, about 10% more, and from 867 to 1,422 on the mixed set. Tokens per correct answer on the multi-hop set land at 0.73x the iterative-RAG baseline, beside the 12B's 0.66x. The harness bounced eleven answers across the two sets for a weak proof or an uncited node, and the model fixed every one of them within its budget.

We are not publishing latency for this run. The model was served over the network by a shared homelab box that was busy with other work, so the wall-clock we saw describes that box, not the model.

The one miss

Fork-tier question 7 asks two things at once: which sales region billed the most, and which product line generated the most support tickets. Those are two SQL aggregates over two different datasets. The 2B found both datasets, ran both queries correctly, and answered correctly at step 5: Northeast, Wand.

Then the harness rejected the answer. Its proof audit wants a sentence copied verbatim from a tool result, and a SQL result set has no sentence in it. The model went back to the sales dataset, re-opened it, re-ran the same query, and ran out of steps without answering again. The navigation was right; the grounding gate was the failure. It is a different failure from the paper's solo miss on the same set, which ran out of steps mid-chain on a width-3 question, and both are in the report rather than out of it.

The fix we had to make, and why it is in the report

On the small Phase-0 fixture the 2B first scored 7/10, and all three misses had the same shape: zero tool calls. The model was reasoning correctly and then writing the call in the XML-like function syntax it was trained on instead of the JSON the prompt asks for. The harness read that as an invalid reply, said so, and the model repeated the syntax until its steps were gone.

The harness now translates that syntax into the same action, with each argument typed by the engine's own signature table. It adds no navigation help. With nothing else changed, the fixture went to 10/10 on two consecutive runs. We are telling you because a format mismatch scores exactly like a wrong answer, and a benchmark that hides that is asking you to trust the wrong thing.

Why this matters

The paper's claim is that grounding is a property of the environment, not of the model. A 12B beating a frontier-model architecture on the same corpus was the first piece of evidence. A 2.5B model matching that 12B on the same forest, with less retrieval help, is the second. Spend intelligence on the environment so you can spend less on the model, and the model you need keeps shrinking.

It also changes what self-hosted costs. A 12B wanted a 12 GB card, the one we benchmarked on. A 2B in 3.3 GB fits a 4 GB card, an Apple Silicon laptop, or a CPU-only box with some patience. The Models screen binds any OpenAI-compatible endpoint, so trying it is a configuration edit: pull the model in Ollama, point the chat binding at it, and run the benchmark yourself.

The usual honesty applies. One run per set, a benchmark generated by us, eleven strictly multi-hop questions, and a preprint. The corpus, the question sets, the harness and the parser change are all committed, and every table regenerates with one command.

Run it on yours

Pull hf.co/openbmb/MiniCPM5-2B-GGUF:Q8_0 in Ollama, set the chat endpoint, build the bench forest, and run the strictly multi-hop set. The exact commands, and the environment lines the Station reads, are in the deploy docs. Star MonkeyLLM on GitHub if the result earned it, and read the preprint at monkeyllm.com.