Do You Need Embeddings? A Decision Guide

"Should we stand up an embedding service?" is an infrastructure decision most teams make by default rather than by evidence. Here are the signals pointing each way, and a test you can run on your own corpus in an afternoon, before any invoice arrives.

Signals that BM25 over curated metadata is enough

Your queries name things. Services, error codes, project names, people, SKUs. Lexical search is strongest exactly where queries contain names, and inside a company most of them do.

You have curated metadata, or can produce it. Titles, aliases and short summaries give BM25 something dense to grip. In our measurement, weighting curated fields (title, aliases) closed the gap to hybrid search entirely: recall@5 = 1.00 at 1.3 ms p95, BM25 over SQLite, no embeddings, no vector database, no GPU.

The caller is an agent, not a person. Agents reformulate and retry at negligible cost, and in an entry-then-walk architecture search only has to open the right door; the walk across linked nodes does the rest.

One dominant language across both corpus and queries.

Signals that you want vectors

Humans typing paraphrases. End users describe rather than name, and share little vocabulary with your documents.

Cross-lingual retrieval. The query arrives in one language, the documents live in another.

No metadata and no write-time budget. Raw chunked bodies with no titles worth weighting leave lexical search nothing to grip.

Vocabulary drift you cannot alias away. A domain where the same concept gets named ten ways by ten teams.

When several of these hold at once, embeddings are not overhead. They are the correct tool, and pretending otherwise would be pretending our workload is everyone's workload.

The test: an afternoon, not a migration

1. Build a gold set. Fifty real queries, pulled from search logs or from teammates, each paired with the document that should answer it.
2. Run the lexical search you already have. SQLite FTS5 or Postgres tsvector, one evening of setup. Weight title and alias fields above body text; in our measurement, that weighting was the entire difference.
3. Measure recall@5. If the target document shows up in the top five nearly every time, an embedder would be improving a number with almost no room left at the entry stage.
4. Only then trial hybrid. Add an embedding model, rerun the same gold set, and price the measured delta against the standing infrastructure it requires: the service, the index, the re-embedding pipeline.

One honesty note: our numbers come from our corpus and our question sets, and we publish them as such, harness included. The claim is not that your corpus will hit 1.00. The claim is that this is measurable on your own data before any money moves.

Where MonkeyLLM sits

The default is the cheap end. Entry search is BM25 over SQLite, weighted on each node's passport (a curated card: title, summary, tags), and those passports are produced automatically at ingest: 1.71 s per document, with 100% of summaries passing a sixty-token contract. The embedder is optional and supported, not removed: point a model binding at any OpenAI-compatible embedding endpoint and entry search becomes hybrid, with no schema change.

The measurement behind that default, and the longer argument, is in BM25 over SQLite vs a vector database.

Run it on your corpus

The gold-set test costs an afternoon and settles the argument with your own data. If the lexical number holds, deploy a forest in four commands, keep everything on your own hardware, and skip the embedding line item until it earns its place. Code at github.com/JimmyWesley/MonkeyLLM, everything else at monkeyllm.com.