Your own bucket (S3, R2, B2, MinIO)

Your data can stay in your bucket. SkimmerDB reads files from it and writes files into it with keys you pass in the request, on AWS S3 or on any S3-compatible service:

• Import files from a key, a prefix or a pattern: POST /v1/namespaces/{name}/import/bucket.
• Deliver exports of a SQL answer as CSV, NDJSON or Parquet: POST /v1/sql/exports with a destination.
• Sync a DuckLake table whose data lives in the bucket: POST /v1/namespaces/{name}/lake/sync.

All three take the same s3 object:

"s3": {
  "endpoint": "https://<account-id>.r2.cloudflarestorage.com",
  "region": "auto",
  "access_key_id": "...",
  "secret_access_key": "...",
  "session_token": ""
}

Leave endpoint out for AWS S3 and give the bucket's region. Leave the keys out for a public bucket (such as AWS Open Data): requests then go unsigned, for imports only. For anything else, endpoint is the service's address: scheme and host, with an optional port and no path. Requests are path-style (https://endpoint/bucket/key) and signed with AWS Signature Version 4.

#Import files from your bucket

curl -s https://api.skimmerdb.com/v1/namespaces/events/import/bucket \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{
    "path": "s3://events/2026/10/*.parquet",
    "id": "event_id",
    "text": ["message"],
    "s3": {"endpoint": "https://<account-id>.r2.cloudflarestorage.com", "region": "auto",
           "access_key_id": "...", "secret_access_key": "..."}
  }'

import os
from skimmerdb import SkimmerDB

db = SkimmerDB()
r2 = {
    "endpoint": "https://<account-id>.r2.cloudflarestorage.com",
    "region": "auto",
    "access_key_id": os.environ["R2_ACCESS_KEY_ID"],
    "secret_access_key": os.environ["R2_SECRET_ACCESS_KEY"],
}
answers = db.namespace("events").import_bucket(
    "s3://events/2026/10/*.parquet", r2, id="event_id", text=["message"]
)
print(sum(a.rows for a in answers), "rows")

import { SkimmerDB } from "skimmerdb";

const db = new SkimmerDB();
const r2 = {
  endpoint: "https://<account-id>.r2.cloudflarestorage.com",
  region: "auto",
  access_key_id: process.env.R2_ACCESS_KEY_ID!,
  secret_access_key: process.env.R2_SECRET_ACCESS_KEY!,
};
const answers = await db.namespace("events").importBucket("s3://events/2026/10/*.parquet", r2, {
  id: "event_id",
  text: ["message"],
});

path is one of these:

• One file: s3://bucket/exports/orders.csv.
• Every file under a prefix: s3://bucket/exports/.
• A pattern: s3://bucket/exports/*.parquet matches the files under exports/ whose names end in .parquet. One *, in the last part of the path.

The format comes from each file's extension (.csv, .ndjson, .jsonl or .json, .parquet) unless you give format. id, text and schema work as they do for an upload.

The answer lists the files imported:

{"files": [{"key": "2026/10/01.parquet", "bytes": 48210331, "rows": 1000000, "written": 1000000}],
 "written": 1000000, "rows": 1000000, "bytes": 48210331, "next": "", "took_ms": 2140, "created": true}

• In key order, in pages. One request takes up to 100 files and 4 GB, and each file can be up to 2 GB. When more are left, next names the last key taken: pass it as after to go on. The Python and TypeScript helpers do that for you until next is empty.
• Ids. With id, each row's value names its document. Without it, a row is <file key>:<row number> (from 0), so importing the same file again replaces its documents instead of adding them twice.
• Each file is its own import. All of a file's rows become visible together. If a file fails (a bad row, a read error), the request stops. The error answer lists the files already in, failed (the key that failed) and next (pass it as after to retry from that file). Ids make a rerun safe: a document written again replaces itself.
• The next file downloads while the current one is parsed, over connections kept open.
• Metered as writes of the documents and the files' bytes, like an upload.

#Deliver exports to your bucket

Add destination to an export. The file goes into your bucket by multipart upload (parts of 32 MB). No copy is kept on our side, so there's nothing to download and nothing expires.

curl -s https://api.skimmerdb.com/v1/sql/exports -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d '{
    "sql": "SELECT * FROM events WHERE day >= '\''2026-10-01'\''",
    "format": "parquet",
    "destination": {
      "path": "s3://analytics/exports/events-october.parquet",
      "s3": {"endpoint": "https://<account-id>.r2.cloudflarestorage.com", "region": "auto",
             "access_key_id": "...", "secret_access_key": "..."}
    }
  }'

e = db.wait_export(db.export(
    "SELECT * FROM events WHERE day >= '2026-10-01'", format="parquet",
    destination="s3://analytics/exports/events-october.parquet", s3=r2,
).id)
print(e.destination, e.bytes)

const e = await db.waitExport((await db.export(
  "SELECT * FROM events WHERE day >= '2026-10-01'", "parquet",
  { path: "s3://analytics/exports/events-october.parquet", s3: r2 },
)).id);

• A path ending in / gets a name made for it: export-<time>.<format>.
• Checked before the job starts. The export writes a small object next to the file and deletes it, so wrong keys, a missing bucket or a key without write access fail at once (422 with the service's error code, such as AccessDenied), not after the query has run.
• The keys stay in memory only while the job runs. If the job fails, the multipart upload is aborted and nothing is left in the bucket.
• The export's status says destination (the file's s3:// path) in place of download.
• A delivered export's bytes count as bytes returned, like a download. On the free plan, an answer bigger than what's left this month fails.

#Credentials and safety

• Never stored. The keys are used for the request, or for the export while it runs, and are then dropped. Use keys that can only reach that bucket: read-only for imports and DuckLake, write (PutObject, multipart, DeleteObject for the check) for exports. R2's "Object Read only" and "Object Read & Write" tokens scoped to one bucket fit.
• Public addresses only. The endpoint's host must resolve to a public address. Private, loopback, link-local and cloud metadata addresses are refused. The connection goes to the address that was checked (no second lookup), with TLS verified against the host's name, over https on port 443 or above 1023.
• Errors carry the code only. When the service refuses a request, the error says the HTTP status and S3's error code (NoSuchKey, AccessDenied, SignatureDoesNotMatch), never the body of the service's answer.