Inside the 11-Question Benchmark
Eleven questions, each needing at least three chained hops. The same 12B model scores 0/11 as a top-k RAG reader and 11/11 as a forest navigator. Here is how the benchmark was built and how to rerun it.
4 min read
Blog
Notes from the forest
Engineering notes on navigation, ingestion, agent memory and honest benchmarking, written by the people who ran the experiments.
Our benchmark has a failure section. On purpose.
The trail-learning convergence criterion was not met, and it is in the report anyway. A benchmark you can only pass is not a benchmark.
3 min read
Your agentic RAG loop answers slower and spends more per correct answer. We measured it.
Iterative RAG loops hide their cost in the wrong denominator. Measured per correct answer: 0.58x the tokens and 8.4 s p95 vs 17.5 s, same 12B model.
3 min read
Want the long version?
The paper carries the full architecture, the benchmark tables and the findings that failed their criteria.