Ingesting 100 Real Documents: The Write-Time Trade

Retrieval pipelines pay their tax at read time, on every query, forever. MonkeyLLM moves that cost to write time, once per document, in a batch you can watch finish. Here is what the ingest pipeline actually does to each document, and what it cost across 100 real ones.

What happens to one document

Every document that enters the forest becomes a node with a curated passport: a title, a summary and tags (the passport is the small label an agent reads before deciding whether to open the full node). The pipeline drafts that passport, wires the node into the graph with links, and commits it. In forest terms the move is called Plant: new knowledge committed as new nodes.

All of this ceremony exists for the read path. Agents navigate by reading labels and following links, opening only what they need, so the label is the product. A sloppy summary at write time becomes a wasted hop on every future walk.

The sixty-token contract

The summary in every passport is held to a contract: sixty tokens. Not "roughly a paragraph", a checkable limit the pipeline enforces and reports. Across our ingest run of 100 heterogeneous real-world documents, 100% of summaries passed the sixty-token contract. When the batch reports done, every label in the forest is one an agent can afford to read on a budget.

Zero broken links

The second post-condition is structural: zero broken links after the batch. Every edge created during ingest resolves to a node that exists. That sounds like a low bar until you have watched an agent walk into a dead reference mid-hunt; a forest where every path leads somewhere is what makes budgeted navigation predictable instead of hopeful.

1.71 seconds per document, measured

The full treatment (passport drafting, linking, verification) costs 1.71 s per document, measured across those same 100 documents on our single RTX 3060 rig.

Now compare the shapes of the two costs. Ingest is a batch: bounded, visible, done once. The Studio's Ingest screen (this post's cover) renders it as a job you can watch, files, folders and clips moving through one by one. Read-time cost is a meter that never stops running. Paying 1.71 s per document once is what lets entry search stay at BM25 over SQLite, at recall@5 = 1.00 and 1.3 ms p95, with no embeddings, no vector database and no GPU on the query path. The Forest Principle essay makes this argument in general form; this post is the same argument in numbers.

The numbers regenerate

100 documents is a real corpus, not a large one, and we say so. The corpus, question sets and harness are committed, and every table in the report regenerates with one command, the ingest table included. Your documents are almost certainly weirder than ours; the honest test is to run the pipeline on them and see whether the contract still holds. The primitives behind nodes, passports and links are described in the primitives docs.

Feed it something real

Deploy with four commands from the quickstart, point an ingest job at your ugliest folder, and watch the batch run. Then reproduce our table from github.com/JimmyWesley/MonkeyLLM, or start with the tour at monkeyllm.com.