DuckLake destination

DuckLake is an open lakehouse format that stores metadata in a SQL catalog and table data in object storage. Supabase Pipelines keeps current-state DuckLake tables synchronized with published Postgres tables.

Understand the DuckLake components#

A DuckLake destination has three independent components:

Always query through the DuckLake catalog. Reading the Parquet objects directly can miss inlined changes, delete files, and the current snapshot selected by the catalog.

You can query replicated tables, but treat them and their underlying catalog and object-storage state as read-only. Writes outside Pipelines can conflict with replication and background maintenance.

Choose a configuration mode#

The Dashboard offers two ways to provide the catalog and storage.

Use Supabase#

Use this mode to back the DuckLake with Supabase projects. Pipelines provisions its own catalog and object-storage credentials when you create the destination.

Before you begin:

• Choose active, healthy, non-branch projects from the same organization for the catalog and storage. You can use the same project for both.
• Make sure your organization role can administer SQL in the catalog project and Storage in the storage project.
• Create a private standard Storage bucket, or create one from the destination form.
• Choose a metadata schema unique to this DuckLake. Use only letters, numbers, and underscores.
• Keep the catalog project and storage project in the same region when possible. Managed Pipelines run in AWS eu-central-1 (Frankfurt), so resources near Frankfurt reduce network latency.

To configure the destination:

1. Open Database > Replication
2. Click Add destination
3. Select DuckLake. If it isn't available, request Early Access.
4. Select a Postgres publication and enter a destination name
5. Select Use Supabase
6. Configure the catalog:
   • Catalog project: The project whose Postgres database stores DuckLake metadata
   • Pool size: The number of concurrent DuckDB connections to the catalog, from 1 to 6. The default is 4.
   • Metadata schema: A unique Postgres schema for this DuckLake's metadata
7. Configure object storage:
   • Storage project: The project whose Storage service holds DuckLake files
   • Bucket: A dedicated private bucket for the DuckLake
8. Click Create and start pipeline

Validation can show warnings that Supabase catalog and Storage credentials are provisioned only when the destination is saved. This is expected for Use Supabase mode. Review the selected projects and proceed to create the destination.

Pipelines-generated credentials remain hidden and are only for the managed writer. Create separate read credentials when connecting an external query engine.

Custom parameters#

Use this mode with a Postgres catalog and S3-compatible object storage that you control.

Prepare the following resources:

1. A Postgres database reachable from managed Pipelines. Create a dedicated user that can create and modify the DuckLake metadata schema and its tables.
2. An S3-compatible bucket and a dedicated prefix for this DuckLake.
3. Object-storage credentials that can list, read, write, and delete objects under that prefix. Delete access is required for managed file cleanup.

Use a new catalog metadata schema and data prefix for each destination. Reusing an existing schema or prefix can mix the metadata or files of different DuckLakes.

Configure these fields in the destination form:

• Catalog URL: A postgres:// or postgresql:// connection URL, including credentials and an sslmode appropriate for your provider
• Data path: An s3://<bucket>/<prefix> URL
• Pool size: From 1 to 6; the default is 4
• S3 access key ID and S3 secret access key: A credential pair for the data path
• S3 region: The storage provider's region
• S3 endpoint: The provider endpoint without http:// or https://
• S3 URL style: path for Supabase Storage and many S3-compatible providers, or vhost for virtual-host-style addressing
• Use SSL: Keep enabled for production endpoints
• Metadata schema: A unique Postgres schema for DuckLake metadata, using only letters, numbers, and underscores

The catalog URL and storage credentials are stored as secrets and aren't returned after creation. When editing the destination, leave a secret field empty to keep its stored value, or enter a new value to replace it.

Query the destination#

Pipelines manages replication but doesn't provide query compute for DuckLake during Early Access. Connect a DuckLake-compatible engine, such as DuckDB with its ducklake extension, to the same Postgres catalog and object-storage path.

For Use Supabase mode, create separate read credentials for the selected catalog and Storage projects. The writer credentials generated for Pipelines aren't exposed. For Custom parameters, use separate read-only credentials when your catalog and storage provider support them.

After attaching the catalog under an alias such as my_ducklake, source schemas and tables are available as qualified DuckLake tables:

1
select *
2
from my_ducklake.public.orders;

See the DuckDB connection guide for the current ducklake extension and ATTACH syntax. If your query client can't read a Supabase-backed catalog during Early Access, contact support.

How replication works#

For each published Postgres table, Pipelines:

1. Creates the corresponding schema and table in DuckLake
2. Copies existing rows during the initial sync
3. Applies subsequent inserts, updates, deletes, and truncates to the current-state table
4. Applies supported source schema changes

Source schema and table names are preserved, while ASCII uppercase letters in column names are converted to lowercase. DuckDB compares schema, table, and column identifiers without ASCII case distinctions, so don't publish names that differ only by ASCII case. Use distinct lowercase column names. Source column names matching the generated supabase_etl_ducklake_dropped_<ordinal>_<hash> shape are reserved for schema-change recovery.

A source TRUNCATE truncates the DuckLake table. Resetting a table drops and recreates the DuckLake table before running a new initial sync. Removing a source table from the publication stops new changes after the pipeline restarts but leaves the destination table in place.

Source table requirements#

Insert-only tables don't require a primary key or replica identity. Updates and deletes require a published Postgres row identity.

Changing replica identity only affects new WAL records. A retained update or delete written before the change can continue to fail after restart and may require recreating the pipeline or restarting the affected table's initial sync.

Type mapping#

Pipelines creates DuckLake columns with these mappings:

DuckDB decimals support precision from 1 to 38 and a scale between 0 and the precision. Postgres numeric values declared outside that range, unconstrained numeric, and numeric types with unsupported modifiers are stored as varchar to preserve their serialized value.

Schema change support#

DuckLake schema change support is limited during Early Access.

Supported changes include:

• Adding columns
• Renaming columns
• Dropping columns
• Dropping NOT NULL from an existing column
• Adding, changing, or removing supported column defaults

The following changes are not applied automatically:

• Changing a column's data type
• Adding NOT NULL to an existing nullable column
• Renaming a source table or schema

New columns are created as nullable so existing destination rows remain valid. If a source default can be represented safely, it is stored as DuckLake metadata; unsupported defaults are skipped with a warning. DuckLake applies each planned multi-column schema change in a transaction.

Troubleshooting#

Additional resources#

• DuckLake documentation
• DuckLake architecture
• Monitor pipeline status