Durability
When a write is safe, when it becomes visible, and what you see when a server or object storage fails.
SkimmerDB keeps every namespace in object storage (S3). Servers hold no data of their own: their RAM and local disk are caches, rebuilt from object storage. So the guarantees come down to two rules: a write is acknowledged only once it is stored in object storage, and nothing becomes visible except whole.
Acknowledged means stored
A write (POST .../docs, PATCH .../docs, POST .../delete, an import) answers 2xx only once it is in object storage. A server crashing, restarting or being replaced afterwards loses nothing: the next server to serve the namespace replays what was acknowledged before it answers. In recovery terms, the recovery point objective for acknowledged writes is zero.
Writes from every tenant on a server are committed together every 250 ms, so a write takes about that long to answer.
If a write's outcome is uncertain (the answer is 503 write_unknown, or the connection dropped), it may or may not have been applied. Retry it: writes are by id, so writing the same documents again gives the same result.
When a write is visible
| Where | Visible |
|---|---|
| The server that took the write | At once when it answers |
| Every other server | Within about a second |
With "wait": true | The answer comes only once the write is searchable on that server |
Use "wait": true when your next request reads what you just wrote.
Nothing half-done
- A write is atomic. A request's documents and deletes become visible together or not at all, and so does every row of an import.
- The newest write of an id wins. Writing an id again replaces the whole document; a deleted id stops counting at once.
- Background work is invisible. Compaction merges a namespace's writes into bigger segments and swaps them in in one atomic step. A query sees the namespace before the swap or after it, never in between, and a compaction interrupted by a crash leaves nothing visible behind.
- Cursors survive compaction. A search's
nextcursor goes on where it left off while the namespace is compacted.
Partial, never wrong
When object storage fails for some of a query's reads, or a search reaches its time limit, the answer says so instead of guessing:
| Query | On a failed read or the time limit |
|---|---|
| Search, vector search | Answers what it found with "partial": true (and "timed_out": true at the time limit) |
| Count | A failed read: "partial": true (the count may be low). The time limit: fails with 422 query_timeout |
| SQL | Fails rather than answer from incomplete data |
| Write | 503 unavailable (nothing was written) or write_unknown (retry) |
A complete answer is always a correct one. Treat a partial answer as incomplete, never as "nothing else matched".
When a server dies
- Its namespaces move to the other servers. Until the new server has replayed the dead one's acknowledged writes, nobody writes those namespaces, so nothing is lost or reordered.
- Reads go on from any server, from object storage, with what was acknowledged.
- A fresh server serves within tens of milliseconds of starting; each namespace then opens on its first query, in one object-storage round trip when it's small, or two.
- The first queries after a restart may be slower while caches warm up: each answer's
temperature(warm,localorcold) says how warm the namespace was.
Deleting
POST .../deleteand writes that replace a document stop the old version counting at once; its bytes leave storage when the namespace is next compacted.DELETE /v1/namespaces/{name}deletes the namespace; its documents are deleted from object storage right after.DELETE /v1/tenants/{id}deletes a tenant's namespaces, keys and usage for good.