Your agentic RAG loop answers slower and spends more per correct answer. We measured it.
Iterative RAG loops hide their cost in the wrong denominator. Measured per correct answer: 0.58x the tokens and 8.4 s p95 vs 17.5 s, same 12B model.
When top-k RAG fails on multi-hop questions, the standard fix is a loop: retrieve, reason, reformulate, retrieve again. It does help. It is also expensive in a way that hides well, because we usually measure cost per query instead of cost per correct answer.
We built MonkeyLLM around a different idea (navigation instead of retrieval) and benchmarked both. Here is the head-to-head, architecture vs architecture.
The setup
The benchmark: 11 strictly multi-hop questions, every one requiring at least 3 chained hops across documents. Yes, 11 questions is a small benchmark, and we say so in the paper. Both sides use the same quantized 12B local model on a single RTX 3060 with 12 GB. No cloud calls.
The numbers
- Tokens per correct answer: 0.58x vs the iterative-RAG baseline.
- Latency: 8.4 s p95 vs the baseline's 17.5 s.
- For accuracy context: that same model scored 0/11 as a classic top-k reader and 11/11 as a forest navigator.
The line we keep coming back to: failing cheaply is not economy. A loop that burns fewer tokens on a wrong answer still bought you a wrong answer.
Why the loop pays more
An iterative loop re-queries an undifferentiated pile of chunks, and every round re-stuffs the context window with overlapping evidence the model must re-read.
MonkeyLLM structures the corpus as a forest of nodes, each with a curated passport (title, summary, tags: a librarian's label). The agent, the monkey, enters through search and walks: read the label, follow a link, open only what it needs. Ten typed tools over MCP, every hop cheap and token-budgeted.
Entry is BM25 over SQLite: recall@5 = 1.00 at 1.3 ms p95. No embeddings, no vector database, no GPU. Weighting the ranking toward curated fields closed the recall gap to hybrid search entirely.
The cost moved, it did not vanish. Ingest runs at 1.71 s per document (measured across 100 heterogeneous real-world documents, 100% of generated summaries passing a sixty-token contract, zero broken links afterwards). You pay once at write time instead of on every query forever.
What did not work
Our trail-learning convergence criterion was not met: hops fell by roughly half the threshold we set. We published the miss instead of dropping it. A benchmark you can only pass is not a benchmark.
Reproduce it
Corpus, question sets and harness are committed to the repo, and every table regenerates with one command. The paper is a preprint, and we say so.
Star the repo, read the preprint, or rerun the whole thing on your own corpus: