Our benchmark has a failure section. On purpose.
The trail-learning convergence criterion was not met, and it is in the report anyway. A benchmark you can only pass is not a benchmark.
We recently published the preprint for MonkeyLLM, an open-source, self-hosted "knowledge forest" that AI agents navigate over MCP instead of a vector database they query. In plain English: the corpus is a graph of nodes, each carrying a curated passport (title, summary, tags), and the agent enters through search and walks, reading labels, following links, opening only what it needs.
This post is not about the wins. It is about why the report also contains a section we could have quietly deleted.
The numbers a marketer would love
On 11 strictly multi-hop questions, where every question needs at least 3 chained hops, the same 12B local model scored 0/11 as a classic top-k RAG reader and 11/11 as a forest navigator.
It did that at 0.58x the tokens per correct answer of an iterative-RAG baseline, at 8.4 s p95 versus the baseline's 17.5 s. Per correct answer is the unit that matters: failing cheaply is not economy.
Entry search is BM25 over SQLite: recall@5 = 1.00 at 1.3 ms p95, with no embeddings, no vector database, no GPU. Weighting the ranking toward curated fields (title, aliases) closed the recall gap to hybrid search entirely.
Ingest runs at 1.71 s per document across 100 heterogeneous real-world documents, with 100% of generated summaries passing a sixty-token contract and zero broken links afterwards. You pay once at write time instead of on every query forever.
The numbers a marketer would hide
Three of them.
First, 11 questions is a small benchmark. Strictly multi-hop questions are expensive to author and verify, but small is small, and we say so.
Second, the paper is a preprint. Not peer reviewed. We say that too.
Third, the one that stings. Successful hunts deposit pheromone traces that mint shortcut links, so paths that work should get easier to find and hop counts should fall over time. We set a convergence criterion for that effect before running the experiment. The criterion was not met: hops fell by roughly half the threshold we set. The section is in the report, written like any other result.
Why publish the miss
Because a benchmark you can only pass is not a benchmark. If the report contained only wins, the wins would be worth less. The failure section is what makes the 11/11 believable.
And because the whole point of self-hosted infrastructure is that you never have to take a vendor's word for anything. So we removed the trust step: the corpus, the question sets and the harness are committed to the repo, and every table in the paper regenerates with one command. The rig is a single RTX 3060 with 12 GB.
Prove us wrong
That is not rhetoric, it is the ask. Clone the repo, run the harness on your own corpus, and if a number does not reproduce, open an issue with the table attached. Falsifiable beats impressive.
Repo: https://github.com/JimmyWesley/MonkeyLLM Site and preprint: https://monkeyllm.com