Here we test Ingot's thesis of structure + similarity search better than just similarity search, and how much does it cost in tokens?
Every column below is answered by this one model through the same agent loop. Only the retrieval tools change.
One embedder across the whole table, including the hosted indexes.
Exact match, decided by code. No partial credit.
Accuracy is the share of attempts where the submitted answer exactly matches the correct one. Code decides it, in src/score/score.ts. No model grades the answers.
The corpus comes from a fixed seed, so the harness knows the true state of the fake project and computes each answer from it. For “which team owns the service with the most open issues since 2026-04-01” it counts open issues per service, takes the top one and looks up its owner: infra.
The model has to answer through a submit_answer tool. Text in the reply doesn’t count.
Both sides are lower-cased and trimmed. A comma-separated string is accepted in place of an array.
By the type of answer, with the rules below.
| Answer type | Correct when |
|---|---|
| Number | Exactly equal. |
| Ordered list | The same items in the same order. |
| Set | Exactly the right items: nothing missing and nothing extra. |
An answer of svc:billing scored as wrong, even though the model had found the right team, infra, on the way. Partial credit is recorded separately as F1 (the “set F1” column) and doesn’t count towards accuracy.
√(p(1−p)/n), where n is the number of attempts. It gets large when n is small: at 6 attempts it’s about ±15–20 points in the middle of the range.This does not test how good models are at taking actions, i.e. sending data somewhere or performing tasks correctly. It only evaluates their ability to provide correct answers based on their recalled knowledge.
An answer counts only when it is exactly right. The bar is overall accuracy; Ctx tokens is the average prompt size, in tokens, the model read to produce each answer.
ingot (via MCP)97%±2%6,319ingot (via API)96%±2%3,815turbopuffer70%±5%39,327vector69%±5%46,732pinecone60%±5%39,274control-same-store-top-k40%±5%37,042raw-context90%±3%33,59415 of 693 agent runs failed outright — the provider threw and nothing was answered: control-same-store-top-k 2 of 99, pinecone 9 of 99, turbopuffer 3 of 99, raw-context 1 of 99. They are scored wrong, because a memory that could not be asked did not answer — but they are infrastructure failures rather than retrieval failures, and a column carrying several of them is reading lower than what it did with the questions it got.
ingot (via MCP) | 0.98 | 99% | 44% | 3.3 | 6,319 | 122 ms | 27.8 s |
|---|---|---|---|---|---|---|---|
control-same-store-top-k | 0.49 | 79% | 6% | 7.6 | 37,042 | 421 ms | 105.8 s |
ingot (via API) | 0.96 | 100% | 39% | 2.8 | 3,815 | 154 ms | 19.7 s |
vector | 0.81 | 93% | 4% | 8.0 | 46,732 | 355 ms | 87.6 s |
pinecone | 0.69 | 88% | 4% | 7.4 | 39,274 | 361 ms | 97.8 s |
turbopuffer | 0.77 | 90% | 4% | 7.4 | 39,327 | 363 ms | 80.7 s |
raw-context | 0.91 | — | — | 0.0 | 33,594 | — | 51.7 s |
Tool-call latency is the time one retrieval call took; run time is the whole answer, the model's own thinking included. Both were measured with 8 agent runs in flight at once, so they compare the columns with each other and not with a run that had the provider to itself.
The same aggregate, ordering, semantic, multi-hop questions, asked as the memory grows from 1× to 16×. The test is done by preloading the memory with varying amounts of history, which is the x axis of the chart. The hypothesis is that more history leads to less accurate recall for naive similarity-based retrieval.
ingot (via API)94%▼35,181 tokens per answervector33%▼1435,997 tokens per answerturbopuffer33%▼1137,365 tokens per answercontrol-same-store-top-k31%▼241,081 tokens per answer · 5 of 36 overflowed| Memory | 1× | 2× | 4× | 6× | 8× | 16× |
|---|---|---|---|---|---|---|
| ingot (via API) | 97%, 5k tokens | 97%, 6k tokens | 97%, 6k tokens | 94%, 7k tokens | 97%, 5k tokens | 94%, 5k tokens |
| vector | 61%, 16k tokens | 56%, 17k tokens | 50%, 21k tokens | 50%, 29k tokens | 47%, 28k tokens | 33%, 36k tokens |
| turbopuffer | 61%, 16k tokens | 50%, 19k tokens | 50%, 21k tokens | 36%, 29k tokens | 44%, 32k tokens | 33%, 37k tokens |
| control-same-store-top-k | 42%, 21k tokens | 39%, 35k tokens | 39%, 28k tokens | 33%, 38k tokens | 33%, 36k tokens | 31%, 41k tokens |
Each point is 18 questions asked 2 times, and the accuracy is averaged over those runs. Runs whose prompt outgrew the model's window are scored wrong: control-same-store-top-k, 1 at 2× and 2 at 4× and 2 at 6× and 4 at 8× and 5 at 16×. Model gpt-5-mini, seed 1. Click a memory to pick it out, or a scale on the axis to see what the memory held at that size.
Exact tool answers leave the model less to reason about. So a small model with Ingot should keep up with a large one that searches a vector store and reads the raw data behind it.
| Metric | gpt-5-mini + ingot (via API) | gpt-5.1 + vector-fetch | Difference |
|---|---|---|---|
| AccuracyHigher is better | 89%8 of 9 | 78%7 of 9 | ▲ +11 pts |
| Tool callsFewer · of 12 | 3.9per answer | 5.9per answer | ▲ 1.5× |
| Ctx tokensFewer · of 15k | 5,559per answer | 11,768per answer | ▲ 2.1× |
| Run timeFaster · of 150s | 28sper answer | 145sper answer | ▲ 5.2× |
Multi-hop only, 9 attempts per cell, 12-call budget, seed 1. At n = 9 the ± is 10–14 points: less of a proof, and more of a guide.
Aggregates, absence, ordering and joins are where structure should tell; the semantic questions are where embeddings should perform better.
| adapter | aggregate | absence | ordering | join | semantic | multi-hop |
|---|---|---|---|---|---|---|
| ingot (via MCP) | 94 | 78 | 100 | 100 | 100 | 100 |
| ingot (via API) | 89 | 100 | 100 | 100 | 89 | 100 |
| turbopuffer | 56 | 89 | 22 | 75 | 94 | 56 |
| vector | 67 | 67 | 33 | 67 | 100 | 56 |
| pinecone | 50 | 89 | 22 | 67 | 83 | 11 |
| control-same-store-top-k | 28 | 67 | 22 | 28 | 89 | 11 |
| raw-context | 67 | 89 | 89 | 100 | 100 | 78 |
A generator builds a world, renders it as the paginated JSON an agent would have received, and computes the gold answer from the world objects. Nothing is annotated by hand. Open a shape to read the questions it asks.
A statistic over the whole corpus, not a lookup.
The answer is defined by what is missing — a field left null, or a record with no counterpart in another tool result. There is no text to be similar to.
Requires a total order, not a neighbourhood.
Records from two or three different tool results, joined on a value nobody declared as a key — a file path in one payload, a service name in another, the team that owns it in a third.
Asked with a paraphrase that shares no distinctive term with the record. The category built to be hard for SQL and easy for embeddings.
Two hops and an argmax.
The corpus is 19 payloads and 501 records of tool output from 6 tools. Nothing declares a key, so the joins run on a file path in one payload and a service name in another.
The initial thesis behind this library was developed on the idea that it could provide better answers when recalling and linking structured tool results, thus we benchmark against JSON tool results from a catelogue.
github.list_pull_requests.files[]catalog.list_files.pathci.list_runs.prgithub.list_pull_requests.numbercatalog.list_files.servicecatalog.list_services.namename, never the ref beside itpagerduty.list_incidents.servicecatalog.list_services.namecatalog, where the service record answers to svc:catalogcatalog.list_servicesOne small JSON pageone payload, 8 records, 556 characters
A service catalogue, arriving whole. It is the only place ownership is recorded, and two of the services record it as null — present and empty, not missing. That makes the absence questions hard without making them unanswerable.
{
"ref": "svc:auth",
"name": "auth",
"owner": null,
"tier": 2
}catalog.list_filesJSON pages of 402 payloads, 40 records a page, 73 records, 3,275 characters in the largest
A repository listing: path, service, size. Nothing an embedding can tell apart — every record reads almost exactly like every other one, so a top-k over them is close to a coin flip.
{
"ref": "file:f-001",
"path": "src/auth/router.ts",
"service": "auth",
"loc": 431
}github.list_pull_requestsJSON pages of 25, nested5 payloads, 25 records a page, 120 records, 8,010 characters in the largest
The largest payloads in the corpus, and the ones with structure inside the structure: each record carries an array of touched files and an array of labels. This is the blob an agent reads once, answers one question from, and drops.
{
"ref": "pr:1400",
"number": 1400,
"title": "document the retry path",
"author": "akiyama",
"state": "open",
"created_at": "2026-05-26T18:00:00.000Z",
"merged_at": null,
"additions": 469,
"deletions": 88,
"files": [
"src/auth/cache.ts",
"src/delivery/config.ts",
"src/notify/client.ts",
"src/search/schema.ts"
],
"labels": []
}ci.list_runsJSON pages of 406 payloads, 40 records a page, 226 records, 6,837 characters in the largest
The most repetitive source, and the biggest by record count: build after build, most of them green and uninteresting until a question is about the one that failed or the three that took longest.
{
"ref": "ci:run-0001",
"id": "run-0001",
"pr": 1400,
"workflow": "unit",
"status": "success",
"duration_sec": 854,
"started_at": "2026-05-26T18:00:00.000Z",
"failed_step": null
}pagerduty.list_incidentsJSON pages of 10, with prose2 payloads, 10 records a page, 14 records, 3,144 characters in the largest
Incidents, each with a written summary — a sentence of English inside a JSON field, which is where the cause of an outage actually lives. The semantic questions ask about these in words the summary never uses.
{
"ref": "inc:INC-01",
"id": "INC-01",
"title": "catalog degraded",
"service": "catalog",
"severity": "sev2",
"started_at": "2026-05-04T18:00:00.000Z",
"resolved_at": "2026-05-04T20:00:00.000Z",
"summary": "Customers saw elevated errors on catalog. The cause was that the connection pool was exhausted under sustained write load."
}linear.search_issuesJSON pages of 20, with free text3 payloads, 20 records a page, 60 records, 5,066 characters in the largest
Issues: a title, a state, an assignee who is sometimes nobody, and a free-text body. The most document-like thing here, and still mostly fields.
{
"ref": "iss:ENG-200",
"id": "ENG-200",
"title": "refactor the retry path in billing",
"service": "billing",
"state": "open",
"assignee": "kowalski",
"created_at": "2026-05-13T17:00:00.000Z",
"body": "Reported against billing. Needs a look before the next release."
}This run is the ordinary corpus — --logs 0 — so every payload above is a paginated listing that fits in a window. The other shape does not: one logs.search that comes back with tens of thousands of lines in a single result. It is a different experiment rather than a bigger one — raw-context is refused before inference rather than scored, and top-k finds a shrinking share of what an aggregate needs while a count over rows does not care how many there are — and no such run is published here yet.
All of the above results are derived from these transcripts.
Load the transcripst to see how the model actually, interacted with the various questions and tools
One agent harness implementation serves every test case, with the same model and settings so results should only differ due to the tool results.
The read surface a real agent gets over MCP: the schema at connect time, SQL, and ranking by meaning.
The same server and the same rows, over the REST API, with the tools written in this repository in the same voice as the baselines’. The gap to `ingot (via MCP)` tells you how much of the result is the surface and how much is the data model.
This is a “control”.Ingot contains a vector index, so a win over a vector store could be the structure — or it could be nothing more than a better chunker. The idea is that this row holds the store constant and takes the structure away: same rows, same vectors, same server, reachable only through top-k semantic search. Whatever separates it from `ingot (via MCP)` is what SQL over typed rows is worth.
The shape of every “just put it in a vector store” answer: embed, rank by cosine, return top-k. Chunked one document per record, so nothing is split mid-object and no chunk mixes two records — the friendliest chunking available. It uses the same embedding model as Ingot in these benchmarks.
The hosted vector database, given the identical embeddings, chunking and search tool as `vector`. It is here to answer the obvious objection that a baseline written in this repository is a strawman: if a production ANN index cannot beat brute-force cosine over the same vectors, then what the top-k rows cannot do belongs to top-k retrieval and not to the baseline. Pinecone’s own embedding models are deliberately not used — one embedder across the whole table is the rule.
The same vectors again, in a hosted index built on object storage. Its full-text index is off: switching it on would make this row a hybrid search while the other two stay dense-only.
No retrieval at all — the whole corpus in the prompt. The ceiling for a memory that fits in the window, and the cost baseline everything else should undercut.
We are publishing a benchmark of our own software. Discount it accordingly, and start here.
A seeded generator builds a world; the corpus is that world rendered as the paginated tool results an agent would have received; the gold answers are computed from the world objects directly. That is what makes hundreds of questions affordable and every run reproducible from a seed. It is also why this is a benchmark of a shape of workload, and not of anyone’s production traffic.
Aggregate questions are scored on the answer alone. A correct count of thirty-seven pull requests is its own evidence, and demanding that thirty-seven records come back through the tools would score the cheapest correct path — one SELECT count(*) — as a total retrieval failure.
Every category is machine-scorable by construction: counts, sets of record ids, ordered lists of record ids. A judge would be a second model whose mistakes land in the same column as the retrieval failures being measured.
The default world is around five hundred records — small enough that raw-context is a usable ceiling, which is the point of including it. Conclusions about a corpus a thousand times larger are not supported by this.
This is worth exactly as much as your ability to go and check it, so every part of it is one file, linked below by the question it answers. If you want to know whether we shaped the questions to flatter ourselves, you can read the generator.
Every question and every gold answer, computed from the generated world rather than annotated. Nothing here is written by hand.
packages/bench/src/questions/questions.tsThe seeded generator. `--seed` reproduces it exactly, and the tool results the adapters ingest are a lossy view of these objects.
packages/bench/src/corpus/world.tsThe world rendered as tool results: paginated JSON with no schema attached, at the page sizes the APIs it imitates use. Every adapter ingests this identical array, and the samples on this page are records out of it.
packages/bench/src/corpus/stream.tsCounts exact, sets by F1, ordered lists in order. No model grades anything.
packages/bench/src/score/score.tsOne interface, ten implementations. The tools each adapter puts in front of the agent.
packages/bench/src/adaptersOne loop for every column: same model, same budget, same answer channel. Only the tool list differs.
packages/bench/src/agent/loop.tsOr run it yourself against a seed of your own — --dry-run prints every question and every gold answer without spending anything.