Deploy anywhere a container can run

The database, and the coordinator

Postgres holds the catalogue and the overlay — never the Parquet. A row is written to the database when it arrives and folded into a Parquet generation later, so what is in Postgres is the rows that have not been folded yet, plus the manifest saying where the folded ones went.

It is also how the replicas agree. The embedding queue, the receipt queue and the delivery outbox are ordinary tables, claimed with FOR UPDATE SKIP LOCKED under a lease, and each sweep that drains one takes a Postgres advisory lock so that exactly one replica is sweeping while the rest serve traffic. Nothing else in the deployment holds a timer, a journal or a lock.

The migrations in apps/ingot/drizzle define the entire schema.

Where the Parquet goes

We support three drivers: filesystem, a local path; s3, meaning AWS and everything that speaks its protocol — MinIO, R2, Ceph; and gcs, held by a service account. filesystem is not a stub. A single node with a volume, or a bucket something else has mounted, is a perfectly good way to run this.

Which one is in use is declared and never inferred. Leave INGOT_STORAGE unset and Parquet goes to the directory in INGOT_DATA_DIR; set a bucket variable and leave the driver unnamed, and the service refuses to boot rather than ignoring it. The failure that rule exists to prevent is a typo in a variable name producing a service that starts, serves traffic and writes every file to a container’s ephemeral disk, where it survives until the next deploy. Naming a driver without its keys is refused the same way, and the message names all of the missing ones at once rather than the first.

# MinIO, as docker-compose.yml runs it
INGOT_STORAGE=s3
INGOT_S3_BUCKET=ingot
INGOT_S3_ACCESS_KEY_ID=ingot
INGOT_S3_SECRET_ACCESS_KEY=ingotingot
INGOT_S3_ENDPOINT=http://localhost:9000
INGOT_S3_REGION=us-east-1

Expensive bits are opt-in.

Embedding is a per-row cost paid once and a summary is an LLM call paid once per ingot, every time a caller asks for receipt: "full".

They work independantly of each other, but when both enabled, they work well together. Both default to local — deterministic offline stand-ins, so a laptop and the test suite need no network, key or bill. Not appropriate for prod however, so each announces itself at boot.

When using either you must supply the provider credentials and a provider without its credentials refuses to boot.

Per replica, times the replicas

Every queue here is drained by a bounded number of workers at once, and that bound is the only thing between a burst of writes and an unbounded burst of calls at whatever INGOT_EMBEDDER and INGOT_SUMMARISER name. The work itself is safe at any concurrency — a claim leases its rows, so two drains take different ones — so what these numbers protect is a quota and a bill rather than correctness.

They are per replica, and that is the number to think in. A write wakes the workers in its own process and takes no advisory lock; only a sweep does. So what a provider actually sees is the bound times however many pods are running — and the chart’s autoscaler moves that on CPU, which means a write burst adds pods and multiplies the fan-out precisely when load is highest. At the defaults and maxReplicas: 10, an embedder sees twenty concurrent batches.

ingot_embeddings_pending, ingot_receipts_pending and ingot_deliveries_pending say whether a queue is falling behind. Read them with max() and never sum(): they are read out of Postgres at scrape time, so every replica reports the same shared depth and summing multiplies a backlog by the pod count.

# a quota divided by maxReplicas,
# not a number per pod
config:
  background:
    embeddings: 2
    receipts: 2
    deliveries: 6

Get notified when embeddings and/or receipts are ready

You can either collect a receipt by polling the SELECT and /add call handed back or specify a channel to get a notification pushed to.

So an ingot can nominate a target with POST /:account/:ingot/config, and each receipt is pushed as it lands: a webhook, which needs nothing set here, or an rmq queue, which needs a broker. The split is deliberate. An ingot’s owner chooses where among their own things a receipt goes — an endpoint, a queue name — and the operator chooses what this service will connect to at all. A tenant naming a broker URL would be a tenant choosing where this service opens an authenticated connection.

A queue named on a deployment with no broker is refused at the call that configures it, naming the variable, rather than accepted and then failing every delivery afterwards in a worker log the caller cannot see. Webhook endpoints are checked the same way and at the same moment: absolute http/https, no credentials in the URL, and loopback, link-local and private literals refused — this service would be reaching them from inside its own network, on somebody else’s behalf.

Delivery is at least once via an outbox mechanism. The intention to deliver is a row gets written in the same transaction as the receipt it announces, and a worker sends it afterwards. The aim is to make a receipt impossible to announce and then lose, or lose and never announce. ingot_deliveries_pending says whether a receiver is keeping up; ingot_deliveries_abandoned should sit at zero.

# nothing at all is needed for webhooks
# a broker adds the second transport
INGOT_RABBITMQ_URL=
  amqps://user:pass@broker:5671

Metrics Endpoint

OLTP metrics are available on the /metrics endpoint.

The rest of the list