Reproducibility Is a Feature
Corpus, question sets and harness are committed, and every table regenerates with one command. When the headline is 0/11 versus 11/11 for the same model, the only honest answer to suspicion is a rerun.
Every number in the MonkeyLLM report regenerates from the repository with one command. The corpus is committed, the question sets are committed, the harness is committed. That is not diligence for its own sake: it is the pitch.
What ships in the repo
Three things travel together in the GitHub repository: the corpus, the question sets (including the 11 strictly multi-hop questions, each needing at least 3 chained hops), and the harness that runs both pipelines and scores them. There is no private test set, no hand-adjusted table, and no figure in the paper the harness cannot rebuild. Delete our results and they grow back.
The number that demands a rerun
The headline is exactly the kind of result that should make an engineer suspicious: the same 12B local model scores 0/11 as a classic top-k RAG reader and 11/11 as a forest navigator.
We wrote about the mechanism in why multi-hop breaks RAG, but no essay is a substitute for the harness. A swing from zero to perfect earns skepticism, and the only honest answer to skepticism is a command you can run yourself.
The rig is deliberately boring
Everything was measured on a single RTX 3060 with 12 GB; a quantized 12B fits it whole. 8 GB cards work with tighter quantization or CPU offload via llama.cpp, and if you have no GPU at all you can point the chat binding at any OpenAI-compatible endpoint (OpenRouter, llama.cpp, Ollama). A benchmark that only reproduces on a cluster is closer to a press release. This one reproduces on a gaming card.
One command per table
Each table in the report is regenerated by the same entry point that produced it: the multi-hop scores, the token and latency economics, the entry-search figures, and the ingest costs (1.71 s per document across 100 heterogeneous real-world documents, 100% of summaries passing a sixty-token contract, zero broken links afterwards). If a future commit changes a number, the diff shows up where diffs belong: in version control, not quietly in a blog post.
"Run it on your own data" is the pitch
MonkeyLLM is self-hosted, so reproduction does not have to stop at our corpus. Ingest your own documents, plant a forest, and put your own questions to it. The Studio Data screen gives you read-only SQL over dataset nodes, so you can inspect exactly what your agent is standing on when it answers. If navigation only beat retrieval on our hand-picked corpus, that would be worth knowing, and you are fully equipped to find out. That is why "run it on your own data" is the product pitch, not a disclaimer buried in a limitations section.
The honest print
The paper is a preprint, and we say so. 11 questions is small, and we say so. One convergence criterion was not met and stays in the report; the deep dive on why the trails did not converge is its own post. A small benchmark cannot prove much on its own, but a reproducible small benchmark can be rerun, extended and challenged by anyone, and that is the point.
One command away
Star MonkeyLLM on GitHub, regenerate a table, then swap in your own data starting from the quickstart. The preprint is at monkeyllm.com.