Search your DuckLake

If your data is already in a DuckLake (a catalog in Postgres, Parquet files in your S3 bucket), you don't need to export or copy anything. Point a namespace at a table and sync it: every live row becomes a document your agents can search, filter and query with SQL. Run the sync again after the lake changes and only the changes are written: inserts, updates and deletes, including rows still inlined in the catalog.

POST /v1/namespaces/{name}/lake/sync with the catalog's URL, the table and S3 keys that can read the lake's data_path. Write scope. A namespace that doesn't exist is created.

curl -s https://api.skimmerdb.com/v1/namespaces/tickets/lake/sync \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{
    "catalog": "postgres://lake_reader:secret@lake-db.example.com:5432/lake?sslmode=require",
    "table": "main.tickets",
    "text": ["subject", "body"],
    "s3": {"region": "us-east-1", "access_key_id": "AKIA...", "secret_access_key": "..."}
  }'

import os, requests

r = requests.post(
    "https://api.skimmerdb.com/v1/namespaces/tickets/lake/sync",
    headers={"Authorization": f"Bearer {os.environ['SKIMMERDB_KEY']}"},
    json={
        "catalog": os.environ["LAKE_CATALOG_URL"],
        "table": "main.tickets",
        "text": ["subject", "body"],
        "s3": {
            "region": "us-east-1",
            "access_key_id": os.environ["LAKE_AWS_KEY"],
            "secret_access_key": os.environ["LAKE_AWS_SECRET"],
        },
    },
)
print(r.json())

const res = await fetch("https://api.skimmerdb.com/v1/namespaces/tickets/lake/sync", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SKIMMERDB_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    catalog: process.env.LAKE_CATALOG_URL,
    table: "main.tickets",
    text: ["subject", "body"],
    s3: {
      region: "us-east-1",
      access_key_id: process.env.LAKE_AWS_KEY,
      secret_access_key: process.env.LAKE_AWS_SECRET,
    },
  }),
});
console.log(await res.json());

The answer says what the sync did:

{"snapshot": 42, "previous": 40, "full": false, "upserted": 120, "deleted": 3,
 "key": "id", "skipped": [], "took_ms": 380, "created": false}

#How rows become documents

• Id. The table's id column, or the column you name as key. With "key": "rowid" the id is the DuckLake row id (DuckDB's rowid, which an UPDATE keeps). The key should be unique: DuckLake doesn't enforce that, and two live rows with the same key are one document.
• Text. The columns in text become the searchable text, as name: value lines. By default that's every text column.
• Metadata. Every other column, typed (integers, floats, booleans as 0/1, dates, times, text), for filters and SQL. _rowid holds the row id. Nested types and blobs aren't kept yet; their names are listed in skipped.

#Keeping it in step

A sync reads the catalog at a snapshot (the latest, or snapshot) and compares it with the snapshot it synced last time. It writes rows whose version began after that, and deletes rows that were live then and aren't now. Compactions, flushes of inlined data and rewritten delete files don't count as changes. If the table's columns change (a rename, a drop, a new type), or you change key or text, the next sync writes the whole table again and deletes whatever isn't live (full: true).

Run it on a schedule, or from the job that writes the lake, right after its commit. A sync with nothing new returns at once. Two syncs of the same namespace never run together: the second gets a 409.

If a sync stops partway (your plan's limit, a lost connection), what it wrote stays, and the next sync picks up again from the last snapshot that finished.

#Credentials and safety

• The catalog URL and S3 keys are used for that sync only. SkimmerDB doesn't store them. Give it a read-only Postgres role and keys that can only read the lake's bucket.
• The catalog's host must resolve to a public address. Private, loopback and link-local addresses are refused, and the connection goes to the address that was checked.
• Data files are read from S3 by byte range (the Parquet footer, then the columns needed), with your keys only. A data_path that isn't s3:// is refused.

#Limits

• DuckLake 1.0 catalogs in Postgres. SQLite and DuckDB-file catalogs aren't supported.
• AWS S3 only for now: no custom endpoints such as R2 or MinIO.
• Each sync counts as writes of the documents it upserted and deleted, as an import does.