Here we test Ingot's thesis of structure + similarity search better than just similarity search, and how much does it cost in tokens?
Aggregates, absence, ordering and joins are where structure should tell; the semantic questions are where embeddings should perform better.
raw-context, which does no retrievalEvery payload here keeps one shape from its first page to its last, and every page carries byte-identical keys to every other. That is the friendliest assumption on this page — a corpus that never changes shape is a table already — and it is the case this project is most flattered by. The drifted corpus is the same world with that assumption taken away.
ingot (via API)99% ±1%3,828ingot (via MCP)84% ±4%5,291hyperspell80% ±4%96,804turbopuffer73% ±4%45,730vector67% ±5%41,148pinecone67% ±5%41,364control-same-store-top-k25% ±4%23,612raw-context86% ±4%33,251| adapter | aggregate | absence | ordering | join | semantic | multi-hop |
|---|---|---|---|---|---|---|
| ingot (via API) | 100 | 100 | 100 | 100 | 100 | 89 |
| ingot (via MCP) | 61 | 78 | 100 | 89 | 94 | 78 |
| hyperspell | 50 | 100 | 33 | 97 | 94 | 67 |
| turbopuffer | 67 | 78 | 22 | 72 | 100 | 78 |
| vector | 61 | 67 | 11 | 67 | 94 | 78 |
| pinecone | 67 | 89 | 33 | 61 | 94 | 44 |
| control-same-store-top-k | 17 | 33 | 22 | 25 | 44 | 0 |
| raw-context | 72 | 89 | 89 | 94 | 100 | 44 |
| adapter | set F1 | evidence recall | evidence precision | tool calls | context tokens |
|---|---|---|---|---|---|
ingot (via MCP) | 0.85 | 86% | 43% | 3.1 | 5,291 |
control-same-store-top-k | 0.32 | 59% | 5% | 6.1 | 23,612 |
vector | 0.77 | 92% | 4% | 7.4 | 41,148 |
hyperspell | 0.81 | 93% | 4% | 7.6 | 96,804 |
raw-context | 0.89 | — | — | 0.0 | 33,251 |
pinecone | 0.75 | 86% | 4% | 7.3 | 41,364 |
turbopuffer | 0.82 | 92% | 4% | 7.9 | 45,730 |
ingot (via API) | 0.99 | 97% | 48% | 2.8 | 3,828 |
12 of 792 agent runs failed outright — the provider threw and nothing was answered: control-same-store-top-k 5 of 99, raw-context 2 of 99, pinecone 5 of 99. They are scored wrong, because a memory that could not be asked did not answer — but they are infrastructure failures rather than retrieval failures, and a column carrying several of them is reading lower than what it did with the questions it got.
Every number above is a mean over transcripts, and a transcript can be checked where a mean has to be trusted. Load them to see what each column actually did to answer a question — the SQL it wrote or the searches it ran, the rows that came back, and the answer it gave.
The initial thesis behind this library was developed on the idea that it could provide betteranswers when recalling and linking structured tool results,thus we benchmark against JSON tool results from a catelogue.
github.list_pull_requests.files[]catalog.list_files.pathci.list_runs.prgithub.list_pull_requests.numbercatalog.list_files.servicecatalog.list_services.namename, never the ref beside itpagerduty.list_incidents.servicecatalog.list_services.namecatalog, where the service record answers to svc:catalogcatalog.list_servicesOne small JSON pageone payload, 8 records, 556 characters
A service catalogue, arriving whole. It is the only place ownership is recorded, and two of the services record it as null — present and empty, not missing. That makes the absence questions hard without making them unanswerable.
{
"ref": "svc:auth",
"name": "auth",
"owner": null,
"tier": 2
}catalog.list_filesJSON pages of 402 payloads, 40 records a page, 73 records, 3,275 characters in the largest
A repository listing: path, service, size. Nothing an embedding can tell apart — every record reads almost exactly like every other one, so a top-k over them is close to a coin flip.
{
"ref": "file:f-001",
"path": "src/auth/router.ts",
"service": "auth",
"loc": 431
}github.list_pull_requestsJSON pages of 25, nested5 payloads, 25 records a page, 120 records, 8,010 characters in the largest
The largest payloads in the corpus, and the ones with structure inside the structure: each record carries an array of touched files and an array of labels. This is the blob an agent reads once, answers one question from, and drops.
{
"ref": "pr:1400",
"number": 1400,
"title": "document the retry path",
"author": "akiyama",
"state": "open",
"created_at": "2026-05-26T18:00:00.000Z",
"merged_at": null,
"additions": 469,
"deletions": 88,
"files": [
"src/auth/cache.ts",
"src/delivery/config.ts",
"src/notify/client.ts",
"src/search/schema.ts"
],
"labels": []
}ci.list_runsJSON pages of 406 payloads, 40 records a page, 226 records, 6,837 characters in the largest
The most repetitive source, and the biggest by record count: build after build, most of them green and uninteresting until a question is about the one that failed or the three that took longest.
{
"ref": "ci:run-0001",
"id": "run-0001",
"pr": 1400,
"workflow": "unit",
"status": "success",
"duration_sec": 854,
"started_at": "2026-05-26T18:00:00.000Z",
"failed_step": null
}pagerduty.list_incidentsJSON pages of 10, with prose2 payloads, 10 records a page, 14 records, 3,144 characters in the largest
Incidents, each with a written summary — a sentence of English inside a JSON field, which is where the cause of an outage actually lives. The semantic questions ask about these in words the summary never uses.
{
"ref": "inc:INC-01",
"id": "INC-01",
"title": "catalog degraded",
"service": "catalog",
"severity": "sev2",
"started_at": "2026-05-04T18:00:00.000Z",
"resolved_at": "2026-05-04T20:00:00.000Z",
"summary": "Customers saw elevated errors on catalog. The cause was that the connection pool was exhausted under sustained write load."
}linear.search_issuesJSON pages of 20, with free text3 payloads, 20 records a page, 60 records, 5,066 characters in the largest
Issues: a title, a state, an assignee who is sometimes nobody, and a free-text body. The most document-like thing here, and still mostly fields.
{
"ref": "iss:ENG-200",
"id": "ENG-200",
"title": "refactor the retry path in billing",
"service": "billing",
"state": "open",
"assignee": "kowalski",
"created_at": "2026-05-13T17:00:00.000Z",
"body": "Reported against billing. Needs a look before the next release."
}This run is the ordinary corpus — --logs 0 — so every payload above is a paginated listing that fits in a window. The other shape does not: one logs.search that comes back with tens of thousands of lines in a single result. It is a different experiment rather than a bigger one — raw-context is refused before inference rather than scored, and top-k finds a shrinking share of what an aggregate needs while a count over rows does not care how many there are — and no such run is published here yet.
Every payload above also keeps one shape from first page to last, which is the friendliest assumption on this page: real tools rename fields, change units, and return an object where a string used to be. --drift is the run that does all of that, and it is the one where committing to a column mapping before the last page has a price — so it costs Ingot more than it costs a vector index. Switch the corpus at the top of this page to read it.
One agent harness implementation serves every test case, with the same model, the same tool-call budget and the same answer channel. Only the retrieval tools differ, so a gap between two columnsshould only be down to the tool results.
The read surface a real agent gets over MCP: the schema at connect time, SQL, and ranking by meaning.
The same server and the same rows, over the REST API, with the tools written in this repository in the same voice as the baselines’. The gap to `ingot (via MCP)` tells you how much of the result is the surface and how much is the data model.
The control, and the most important column on this page. It is the sceptic’s question, run rather than argued. Ingot contains a vector index, so a win over a vector store could be the structure — or it could be nothing more than a better chunker. This row holds the store constant and takes the structure away: same rows, same vectors, same server, reachable only through top-k semantic search. Whatever separates it from `ingot (via MCP)` is what SQL over typed rows is worth, and nothing else.
The shape of every “just put it in a vector store” answer: embed, rank by cosine, return top-k. Chunked one document per record, so nothing is split mid-object and no chunk mixes two records — the friendliest chunking available, given deliberately. Same embedding model as Ingot, and brute-force exact cosine rather than an approximate index. What it cannot do is a property of top-k retrieval, not of a baseline built to lose.
The hosted vector database, given the identical embeddings, chunking and search tool as `vector`. It is here to answer the obvious objection that a baseline written in this repository is a strawman: if a production ANN index cannot beat brute-force cosine over the same vectors, then what the top-k rows cannot do belongs to top-k retrieval and not to the baseline. Pinecone’s own embedding models are deliberately not used — one embedder across the whole table is the rule.
The same vectors again, in a hosted index built on object storage. Its full-text index is off: switching it on would make this row a hybrid search while the other two stay dense-only, and hybrid retrieval deserves a column of its own rather than a silent edge in this one.
A hosted memory, configured as its own documentation says to configure it.
No retrieval at all — the whole corpus in the prompt. The ceiling for a memory that fits in the window, and the cost baseline everything else should undercut.
We are publishing a benchmark of our own software, which you should discount accordingly. So we shall attempt to highlight where it is weak.
A seeded generator builds a world; the corpus is that world rendered as the paginated tool results an agent would have received; the gold answers are computed from the world objects directly. That is what makes hundreds of questions affordable and every run reproducible from a seed. It is also why this is a benchmark of a shape of workload, and not of anyone’s production traffic.
Aggregate questions are scored on the answer alone. A correct count of thirty-seven pull requests is its own evidence, and demanding that thirty-seven records come back through the tools would score the cheapest correct path — one SELECT count(*) — as a total retrieval failure.
Every category is machine-scorable by construction: counts, sets of record ids, ordered lists of record ids. A judge would be a second model whose mistakes land in the same column as the retrieval failures being measured.
Ingot asks for a column mapping up front and a vector store does not. Ingestion is timed, but that asymmetry is real and this page does not put a number on it.
`raw-context` reads the whole corpus and answers from it, which makes it the upper bound on what this model does with complete information — but only for as long as the corpus fits in a context window. Above that the request is refused before inference, and a run at that size has no ceiling on the page at all. This one is around five hundred records, well inside the window, so the bound holds here. It would not for a memory a thousand times larger.
Question templates get added to the harness as the workload it models gets better understood, so a published table is a snapshot of the set as it stood on its date. The run id, the seed and the date above pin exactly which questions were asked, and the generator is one link away. But a category is described here by what it is for, which may be broader than the sample any one run drew from it.
The default world is around five hundred records — small enough that raw-context is a usable ceiling, which is the point of including it. Conclusions about a corpus a thousand times larger are not supported by this.
This is worth exactly as much as your ability to go and check it, so every part of it is one file, linked below by the question it answers. If you want to know whether we shaped the questions to flatter ourselves, you can read the generator.
Every question and every gold answer, computed from the generated world rather than annotated. Nothing here is written by hand.
packages/bench/src/questions/questions.tsThe seeded generator. `--seed` reproduces it exactly, and the tool results the adapters ingest are a lossy view of these objects.
packages/bench/src/corpus/world.tsThe world rendered as tool results: paginated JSON with no schema attached, at the page sizes the APIs it imitates use. Every adapter ingests this identical array, and the samples on this page are records out of it.
packages/bench/src/corpus/stream.tsCounts exact, sets by F1, ordered lists in order. No model grades anything.
packages/bench/src/score/score.tsOne interface, ten implementations. The tools each adapter puts in front of the agent.
packages/bench/src/adaptersOne loop for every column: same model, same budget, same answer channel. Only the tool list differs.
packages/bench/src/agent/loop.tsOr run it yourself against a seed of your own — --dry-run prints every question and every gold answer without spending anything.