Ingestion

The four stages#

Every file walks the same pipeline, and each stage has one job. Only stage 2 ever calls a model, and it is the only stage you can skip.

Two properties of that shape are worth stating up front, because they decide what ingestion costs you:

• Conversion needs no model. Adopting a folder of markdown, CSV and JSON works with no MONKEYLLM_LLM_ENDPOINT configured at all. Without a model, summaries are derived from the document's opening sentences and tagged source: ingest at confidence 0.7.
• Nothing fails the batch. A converter that raises, a hook that throws, a model that refuses: each one is recorded in the report and the walk continues. Already-planted nodes stay planted, because each one is its own commit.

What each file becomes#

A converter claims a file by its extension, and what it hands back decides the node's type. Three outcomes exist: markdown (which becomes a note, a document or a media node depending on the source), a declarative dataset the engine builds a SQLite payload from, and a payload the Gardener installs as it stands.

Optional dependencies#

Markdown, text, CSV, JSON, SQLite, images and audio need nothing beyond the base install. SQLite in particular needs no extra: it is the standard library, and it is already the forest's payload format. The office formats need third-party readers, all of them MIT or BSD licensed, shipped together as the ingest extra:

pip install -e ".[ingest]"    # openpyxl, python-docx, xlrd

The built-in list is assembled by import probe, so a missing extra never crashes a sync. The converter simply is not registered, no converter claims the extension, and the file lands in unsupported where you can see it.

Unsupported files and failures#

Nothing is dropped silently. Every file the walk touches ends up in exactly one list of the report:

When more than one converter claims an extension, a failure falls through rather than out: the next claimant in discovery order gets the file, and the fallback is recorded in errors naming who failed on what. Only a failure by the last claimant makes the file terminal. This is what lets a vision describer whose endpoint is down degrade to the media stub instead of aborting the batch.

Datasets and the sample map#

Tabular sources do not become prose. They become real SQLite payloads an agent reads with read-only SQL through the query primitive. Two paths lead there:

• A SQLite file is adopted whole. The converter reads the structure of every table and its first three rows; the Gardener copies the file into place beside the passport. Nothing is rebuilt row by row, so a 5 GB database costs the same to adopt as a 5 MB one, and nothing is lost to a type round trip.
• A CSV, JSON array or workbook is converted into a declarative schema plus rows. The engine generates the CREATE TABLE statements; the converter never writes DDL. Table and column names are slugified into SQL-safe identifiers, so invoices-2026-q1.csv becomes the table invoices_2026_q1.

Either way the passport body carries the sample map: two generated sections that are the only thing any text primitive can see inside a payload.

## Query manual

Tables:
- `invoices_2026_q1(invoice_id TEXT, customer TEXT, issued_on TEXT, total_usd REAL, status TEXT)`

Example queries:
- `SELECT * FROM invoices_2026_q1 LIMIT 5`
- `SELECT COUNT(*) FROM invoices_2026_q1`

Then ## Sample rows. Each sampled table gets its own ### heading naming the table and its row count, followed by up to three rows as a pipe table:

| invoice_id | customer | issued_on | total_usd | status |
| --- | --- | --- | --- | --- |
| INV-1041 | Northwind | 2026-01-04 | 1240.0 | F |
| INV-1042 | Contoso | 2026-01-04 | 880.5 | A |
| INV-1043 | Fabrikam | 2026-01-07 | 15300.0 | F |

The map is deliberately bounded, and every omission is stated in the text rather than left for the reader to discover:

On a table wider than twelve columns the generated example query is SELECT COUNT(*) and a note, not SELECT *: on a wide table the star would come back truncated by the response budget, and offering it would be offering a statement that cannot work.

The Notes section#

The map says what is in a dataset. It cannot say what it means: that one column is USD and another BRL, that status uses one-letter codes, which join answers the question people actually ask. An agent writing SQL without that writes SQL that runs and answers wrongly, which is the worst failure this system can produce, because it looks like success.

So a dataset passport carries a ## Notes section that belongs to a person. The Gardener rewrites ## Query manual and ## Sample rows and only those, section by section, so your notes survive every sync, every re-adoption and every payload replacement. Curation never writes it either: a model's guess about what a column means is exactly what this section exists to correct.

Images and audio#

An image is never unsupported. Images and audio plant as media nodes: the original bytes become the payload, and the body is the textual proxy the forest searches. Text to find, binary to consume.

What that body says depends on what is bound:

• With no vision model, a built-in stub writes what is known without one: the filename, the format and the size, plus the plain admission that no description has been generated yet. The node exists, is findable by its name and its place, and can be described later.
• With a model bound to the vision role, the Station injects a describer ahead of the stub. It is asked to state what the image shows and then transcribe any legible text: labels, headings, code, and the structure of any diagram. The transcription is the load-bearing half, because exact-term search reads the proxy and nothing else, and a slide whose bullet points were never written down is a slide no term search will ever land in.

Curation, the only model stage#

Stage 2 always runs. What runs inside it depends on how much you have wired up, in three layers:

1. 1

   Default tags

   Tags from curation.default_tags in the forest config are merged into every draft. No model, no network.
2. 2

   on_curate hooks

   Deterministic enrichment from installed plugins, run in discovery order. A hook that raises is recorded in errors and the batch continues.
3. 3

   The Curator

   Only with --curate. The model writes the summary and the tags, and proposes links. Everything it produces is validated before it is planted, and every rejection falls back to the derived summary rather than blocking the plant.

The Curator reads the environment, exactly like every other model binding in the engine:

export MONKEYLLM_LLM_ENDPOINT=https://openrouter.ai/api/v1
export MONKEYLLM_LLM_API_KEY=sk-or-...
export MONKEYLLM_LLM_MODEL=google/gemma-3-12b-it
# optional: completion budget, default 300
export MONKEYLLM_LLM_MAX_TOKENS=300

With MONKEYLLM_LLM_MODEL left at its default of local, the client asks the endpoint's /models what it serves and takes the first entry, which is right for a single-model server and wrong for a hosted catalog. Always name the model explicitly with a provider like OpenRouter.

The summary contract#

The summary is the scent: an agent decides from the summary alone whether a node matters. So the Curator is not asked politely, it is held to a contract that is checked in code before the node is planted:

• At most 60 tokens, one to three sentences. The model is given the limit in characters rather than tokens, because characters are something a model can approximate and tokens are not.
• Sentence one says what it is (category plus subject), sentence two the key content: concrete numbers, names, time scope. An optional third says what is not here and where the complement lives.
• Dated content carries its date (at least month and year), because search matches summaries, and a date left in the body is invisible to a time-scoped search.
• No boilerplate openings. A summary starting with "this document describes" or "file containing" is rejected outright: it spends tokens without adding scent.
• Same language as the content, plus up to five tags, lowercase and single words.

A summary that fails is retried, up to three attempts, with the validation error handed back to the model. A summary that fails only on length is trimmed instead of discarded: whole sentences come off first, then words, marked with an ellipsis. Emptiness and boilerplate are not trimmable, because no amount of cutting gives boilerplate a scent, so those fall back to the derived summary and count as a fallback in the run's statistics.

Edge proposals#

With curation on, the Curator also proposes up to three related-to links per node. The candidates come from a closed list the catalog offers, capped at eight, with the node itself and its parent removed. The model picks from that list or picks nothing, so a hallucinated link target is structurally impossible rather than merely unlikely. Picking nothing is a valid and common answer.

Accepted proposals are planted at confidence 0.3, the bottom rung of the confidence ladder. That is deliberate: it is precisely the population the Ranger later promotes or prunes based on real traffic. A link you are certain about is made by editing the node, not by ingest.

The CLI#

Three verbs cover ingestion. All of them operate on an existing forest: a directory without _meta/schema.md is not a forest, and the CLI refuses rather than quietly creating one where you happened to be standing.

vine adopt#

Mirror a source tree into the forest. Directories become branch nodes, files become leaf nodes, and the source root is recorded in _meta/gardener.yaml before the first file is processed, so a run interrupted halfway can be finished by sync rather than started over.

vine adopt <source> [--forest DIR] [--dest BRANCH] [--curate]

The walk skips the usual noise by default (.git, __pycache__, node_modules, _derived, _assets, lock and temp files, .DS_Store) and prunes any nested forest it meets whole: another forest's passports are somebody's curated nodes, not documents to convert.

vine sync#

Re-read the adopted source and reconcile. The forest itself is the sync state, read back from the source_path, source_hash, source_size and source_mtime in each passport, so there is no side ledger to drift out of step. Unchanged size and modification time skip the hash entirely; a changed hash refreshes the body through the same audited write path a human edit uses.

vine sync [source] [--forest DIR] [--curate] [--path REL]

A source file that has disappeared makes its passport stale. It is reported and left alone. Deleting knowledge because a file moved is not a decision an ingest tool gets to make.

vine rollup#

Synthesize branch summaries bottom-up, deepest first, from the entry lines of each branch's children. A curated adopt or sync runs this for you at the end; the standalone verb is for re-running it after hand edits.

vine rollup [--forest DIR] [--all]

By default it only touches branches whose source is ingest, leaving hand-authored regions alone; --all includes them. An empty region keeps its template summary and counts as skipped, and a model failure falls back to a deterministic summary composed from the child titles.

Worked example#

Feed a folder, then ask the resulting dataset a question in SQL. This is the whole loop, from files nobody can search to an answer an agent can quote.

exports/
  master-agreement.pdf
  refund-policy.docx
  whiteboard-pricing.png
  finance/
    crm.db
    invoices-2026-q1.csv

vine init  --forest ./brain --title "Ops brain"
vine adopt ./exports --forest ./brain --curate

curation model: google/gemma-3-12b-it
planted (4):
  finance/crm
  finance/invoices-2026-q1
  refund-policy
  whiteboard-pricing
branches (1):
  finance/_index
unsupported (1):
  master-agreement.pdf
rollup: 1 branch(es) rolled, 0 fallback(s), 1 skipped
curation: {'llm_summaries': 4, 'fallbacks': 0, 'retries': 1, 'skipped': 0, 'links_proposed': 2, 'proposal_fallbacks': 0, 'branch_rollups': 1, 'branch_fallbacks': 0, 'transport_errors': 0, 'rejected': 0, 'repaired': 1}

Read that report line by line, because every list is load bearing:

• Four nodes planted. finance/crm is the database, adopted whole. finance/invoices-2026-q1 is the CSV, converted into a table. refund-policy is the Word document. whiteboard-pricing is a media node.
• One branch created, finance/_index, mirroring the subdirectory. Files at the top of the source hang off the master index directly, which is why the other three have no prefix.
• One file unsupported: the PDF, because no converter claims .pdf until you name one.
• The rollup wrote a summary for the new branch and skipped the master index, which is not an ingest branch.
• Curation wrote four summaries with no fallbacks, needed one retry, repaired one over-long summary by trimming, and planted two proposed links.

The CSV node now holds a real SQLite payload. Ask it something, through MCP, through the engine directly, or over the Station's HTTP surface:

{
  "name": "query",
  "arguments": {
    "id": "finance/invoices-2026-q1",
    "sql": "SELECT customer, ROUND(SUM(total_usd), 2) AS billed FROM invoices_2026_q1 WHERE status = 'F' GROUP BY customer ORDER BY billed DESC LIMIT 3"
  }
}

from monkeyllm import Vine

vine = Vine("./brain", writable=False)
result = vine.query(
    "finance/invoices-2026-q1",
    "SELECT customer, ROUND(SUM(total_usd), 2) AS billed "
    "FROM invoices_2026_q1 WHERE status = 'F' "
    "GROUP BY customer ORDER BY billed DESC LIMIT 3",
)

curl -sX POST https://station.example.com/v1/forests/brain/query \
  -H "Authorization: Bearer $MONKEYLLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "finance/invoices-2026-q1",
       "sql": "SELECT customer, ROUND(SUM(total_usd), 2) AS billed FROM invoices_2026_q1 WHERE status = '"'"'F'"'"' GROUP BY customer ORDER BY billed DESC LIMIT 3"}'

{
  "columns": ["customer", "billed"],
  "rows": [
    ["Fabrikam", 184320.5],
    ["Northwind", 97110.0],
    ["Contoso", 64980.25]
  ],
  "row_count": 3,
  "limited": false,
  "elapsed_ms": 1.87
}

Note what the response contains and what it does not. There is no chunking, no similarity score and no prose: columns, rows, a row count, whether a LIMIT was injected, and how long it took. The engine guarantees the rest:

• Read-only, one statement. The statement must begin with SELECT or WITH; a semicolon inside it is refused, and so is any of ATTACH DETACH PRAGMA INSERT UPDATE DELETE DROP ALTER CREATE VACUUM REINDEX. Writing rows is a different primitive with a different capability.
• Bounded by default. A statement with no LIMIT gets LIMIT 200 appended, and limited in the response tells you when that bound is the reason you got exactly 200 rows.
• Bounded in time. Two seconds, enforced by a progress handler on the connection, not by a wrapper you can escape.
• Truncation is never silence. If whole rows do not fit the response budget, truncated appears with a hint that states, first and in those words, that the missing rows matched the query and exist. A model that reads a display bound as a count reports a wrong answer confidently, so the payload says so before it offers advice.

Configuring the Gardener#

Per-forest configuration lives in _meta/gardener.yaml. It is not a node (it is not markdown), and it is where the Gardener also records the adopted source root and destination.

converters:
  ".pdf": '"/opt/tools/pdf2md" "{input}" "{output}"'
curation:
  default_tags: ["internal"]
  directives: |
    Prefer a formal tone. Flag anything mentioning PII.

Your own converters#

Converters are discovered from three sources, in this order, and the first one that claims an extension wins:

Command hooks are the escape hatch with no license implications: the tool runs in its own process, invoked with {input} and {output} substituted into your template. It must write markdown to the output path (or print it to stdout) and exit zero. A first-line # Title becomes the node title. A non-zero exit or empty output aborts that one file, with the tool's own stderr carried into the report; the rest of the batch is unaffected.

Entry point plugins are the installable route. Register a class or callable implementing convert(self, path: Path) -> Conversion under the monkeyllm.converters group:

[project.entry-points."monkeyllm.converters"]
pdf = "my_monkeyllm_plugin:PdfConverter"

A Conversion carries a kind (markdown, dataset or payload), a title, and then either markdown, or a declarative schema plus rows, or the structure and samples that describe a payload. A plugin that fails to import or raises on load is skipped, never blocking discovery of the rest.

on_curate hooks#

A hook is a callable that takes the frontmatter draft and returns it, mutated or replaced. It runs after the Curator, or instead of it when curation is off, and always before the plant.

def add_compliance_tag(draft: dict) -> dict:
    draft.setdefault("tags", []).append("compliance")
    return draft

[project.entry-points."monkeyllm.hooks"]
on_curate = "my_monkeyllm_plugin:add_compliance_tag"

Hooks run in discovery order. One that raises has its error recorded in the report and nothing else: already-planted nodes stay planted, later files keep processing.

What the Gardener never does#

• It never deletes a node. A source file removed on disk makes its passport stale. The Ranger is the only process that later acts on stale state, and even it never deletes.
• It never writes DDL. Dataset converters hand back a declarative schema and the engine generates the CREATE TABLE statements, so every dataset in a forest was created the same way.
• It never invents an edge target. Proposals are drawn from a closed candidate list of nodes that already exist. There is no id for a model to hallucinate.
• It never overwrites your notes. A sync rewrites the two generated sections of a dataset body and leaves everything else, including ## Notes, exactly as it found it.
• It never plants a broken node. If the passport is refused, an installed payload is removed again. A file that exists and a node that references it are one thing or neither.

Where to go next#