Where the Latency Goes

Same model, same corpus, same 11 questions: the iterative-RAG baseline answers at 17.5 s p95, the forest navigator at 8.4 s. The model did not change and the hardware did not improve. The difference is the shape of the loop, so let's dissect both.

Anatomy of the iterative-RAG loop

An iterative-RAG system runs a cycle: retrieve, reason, reformulate, repeat. Each round costs a retrieval, then a full model pass over a freshly stuffed context, then a decision to stop or to rewrite the query and go again.

Two properties make that loop slow at the tail. First, the loop body is expensive: every round re-stuffs the context with top-k chunks that overlap heavily with the previous round's, and the model re-reads what it already read. Second, the round count is demand-driven: precisely the hard questions, the ones that define your p95, trigger the most reformulations. The tail multiplies an expensive body by its worst-case count.

The navigator's unit of work

The navigator's loop body is a hop: read a passport (the curated title, summary and tags on a node), follow a link, open a full node body only when needed. The reasoning pass per hop runs over a small, budgeted context instead of a chunk pile, so each pass is short.

Entry into the forest is not a bottleneck either. Entry search is BM25 over SQLite: recall@5 = 1.00 at 1.3 ms p95 on this corpus, with no embeddings, no vector database, and no GPU on the search path. Weighting the curated fields (title, aliases) closed the gap to hybrid search entirely, which is why the embedder can stay optional and supported rather than mandatory.

Why the tail collapses

Hard questions cost the navigator extra hops too, but the increments are cheap: a millisecond-scale lookup or link-follow, plus a short budgeted pass over a passport. Variance measured in cheap units keeps p95 close to the median. Variance measured in full retrieve-and-read rounds does not, and that is the entire 17.5 s story.

Budgets add a second, blunter guarantee: every hop is token-budgeted over MCP, so the walk cannot wander indefinitely. The baseline has no such ceiling by construction; you bolt one on with a max-iterations knob, and then the hard questions hit the knob and fail slowly, which is the most expensive way to fail.

The token side of this same comparison, 0.58x per correct answer, is measured in agentic RAG vs budgeted navigation, and the reason single-shot retrieval cannot simply skip the loop is in why multi-hop breaks RAG.

The usual caveats, on purpose

Eleven questions is a small set and we say so. The paper is a preprint and we say so. The corpus, question sets and harness are committed, every table regenerates with one command, and the report keeps a criterion we failed, because numbers you cannot re-derive are marketing.

Time your own tail

Deploy a forest in four commands with the deploy guide, wire your agent over MCP, and measure your own p95. Star or fork the repo, and the rest lives at monkeyllm.com.