# Search your DuckLake

> Keep a DuckLake table searchable in a namespace. Each sync reads your catalog and S3 data where it is and writes only what changed since the last one.

If your data is already in a [DuckLake](https://ducklake.select) (a catalog in Postgres,
Parquet files in your S3 bucket), you don't need to export or copy anything. Point a namespace
at a table and sync it: every live row becomes a document your agents can search, filter and
query with SQL. Run the sync again after the lake changes and only the changes are written:
inserts, updates and deletes, including rows still inlined in the catalog.

`POST /v1/namespaces/{name}/lake/sync` with the catalog's URL, the table and S3 keys that can
read the lake's `data_path`. Write scope. A namespace that doesn't exist is created.

```sh title="curl"
curl -s https://api.skimmerdb.com/v1/namespaces/tickets/lake/sync \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{
    "catalog": "postgres://lake_reader:secret@lake-db.example.com:5432/lake?sslmode=require",
    "table": "main.tickets",
    "text": ["subject", "body"],
    "s3": {"region": "us-east-1", "access_key_id": "AKIA...", "secret_access_key": "..."}
  }'
```

```python title="Python"
import os, requests

r = requests.post(
    "https://api.skimmerdb.com/v1/namespaces/tickets/lake/sync",
    headers={"Authorization": f"Bearer {os.environ['SKIMMERDB_KEY']}"},
    json={
        "catalog": os.environ["LAKE_CATALOG_URL"],
        "table": "main.tickets",
        "text": ["subject", "body"],
        "s3": {
            "region": "us-east-1",
            "access_key_id": os.environ["LAKE_AWS_KEY"],
            "secret_access_key": os.environ["LAKE_AWS_SECRET"],
        },
    },
)
print(r.json())
```

```ts title="TypeScript"
const res = await fetch("https://api.skimmerdb.com/v1/namespaces/tickets/lake/sync", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SKIMMERDB_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    catalog: process.env.LAKE_CATALOG_URL,
    table: "main.tickets",
    text: ["subject", "body"],
    s3: {
      region: "us-east-1",
      access_key_id: process.env.LAKE_AWS_KEY,
      secret_access_key: process.env.LAKE_AWS_SECRET,
    },
  }),
});
console.log(await res.json());
```

The answer says what the sync did:

```json
{"snapshot": 42, "previous": 40, "full": false, "upserted": 120, "deleted": 3,
 "key": "id", "skipped": [], "took_ms": 380, "created": false}
```

## How rows become documents

- **Id.** The table's `id` column, or the column you name as `key`. With `"key": "rowid"`
  the id is the DuckLake row id (DuckDB's `rowid`, which an `UPDATE` keeps). The key should be
  unique: DuckLake doesn't enforce that, and two live rows with the same key are one document.
- **Text.** The columns in `text` become the searchable text, as `name: value` lines. By
  default that's every text column.
- **Metadata.** Every other column, typed (integers, floats, booleans as 0/1, dates, times,
  text), for [filters](/reference/filters) and [SQL](/guides/analytics). `_rowid` holds the
  row id. Nested types and blobs aren't kept yet; their names are listed in `skipped`.

## Keeping it in step

A sync reads the catalog at a snapshot (the latest, or `snapshot`) and compares it with the
snapshot it synced last time. It writes rows whose version began after that, and deletes rows
that were live then and aren't now. Compactions, flushes of inlined data and rewritten delete
files don't count as changes. If the table's columns change (a rename, a drop, a new type), or
you change `key` or `text`, the next sync writes the whole table again and deletes whatever
isn't live (`full: true`).

Run it on a schedule, or from the job that writes the lake, right after its commit. A sync with
nothing new returns at once. Two syncs of the same namespace never run together: the second gets
a 409.

If a sync stops partway (your plan's limit, a lost connection), what it wrote stays, and the
next sync picks up again from the last snapshot that finished.

## Credentials and safety

- The catalog URL and S3 keys are used for that sync only. SkimmerDB doesn't store them.
  Give it a read-only Postgres role and keys that can only read the lake's bucket.
- The catalog's host must resolve to a public address. Private, loopback and link-local
  addresses are refused, and the connection goes to the address that was checked.
- Data files are read from S3 by byte range (the Parquet footer, then the columns needed),
  with your keys only. A `data_path` that isn't `s3://` is refused.

## Limits

- DuckLake 1.0 catalogs in Postgres. SQLite and DuckDB-file catalogs aren't supported.
- AWS S3 only for now: no custom endpoints such as R2 or MinIO.
- Each sync counts as writes of the documents it upserted and deleted, as an
  [import](/guides/import) does.