Use SkimmerDB from DuckDB

DuckDB and SkimmerDB work well together. DuckDB runs analytics where you are; SkimmerDB serves search and SQL to apps and agents. Data moves between them three ways, with no glue code.

#Query SkimmerDB from DuckDB

Both SQL endpoints also answer a GET with the query in ?sql=, so DuckDB's read_csv and read_json can read an answer straight over https. Give DuckDB your key once as an HTTP secret:

CREATE SECRET skimmerdb (
    TYPE http,
    BEARER_TOKEN 'tb_...',
    SCOPE 'https://api.skimmerdb.com'
);

-- an answer as CSV, typed by DuckDB
SELECT kind, n
FROM read_csv('https://api.skimmerdb.com/v1/sql?format=csv&sql=SELECT%20kind%2C%20count(%2A)%20AS%20n%20FROM%20events%20GROUP%20BY%20kind')
ORDER BY n DESC;

-- or as NDJSON, here with a text predicate that runs on SkimmerDB's index
SELECT count(*)
FROM read_json(
    'https://api.skimmerdb.com/v1/sql?format=ndjson&sql=SELECT%20id%2C%20n%20FROM%20events%20WHERE%20contains(text%2C%20%27timeout%27)',
    format = 'newline_delimited'
);

• Encode the SQL. It goes in the URL, so encode it (%20 for spaces and so on). Write * as %2A: DuckDB reads a bare * in a URL as a file glob and refuses it.
• Streamed, every row. format=csv and format=ndjson stream the whole answer. A plain scan of one namespace has no row limit; other query shapes stop at their row cap (at least 1,000,000 rows).
• Reads only. A GET that would write (INSERT, DELETE, CREATE TABLE AS) answers 405; use POST for those.
• /v1/sql sees every namespace of the key's tenant by name. /v1/namespaces/{name}/sql has that namespace as docs.
• Each request is one query for metering and limits, like a POST.

#Send answers to DuckDB as Parquet

For big answers, an export writes a Parquet file. With a destination it goes into your own bucket, on AWS S3, R2, B2 or MinIO, where DuckDB reads it directly:

CREATE SECRET r2 (
    TYPE r2,
    KEY_ID '...',
    SECRET '...',
    ACCOUNT_ID '<account-id>'
);

SELECT status, count(*) FROM 'r2://analytics/exports/tickets.parquet' GROUP BY status;

#Keep a DuckLake table searchable

If your tables are in a DuckLake, sync one into a namespace. Each sync reads the catalog and the Parquet files where they are and writes only the rows that changed. After that, apps and agents can search it by substring, regex, filters and vectors, which DuckDB has no index for.