Re: Enable CDC in Lakeflow Connect Tables

Nmtc9to5 · 3 weeks ago

Hello everyone, I'm implementing a project that requires AutoCDC workflows using Lakeflow.

The architecture is as follows: Data is ingested from a database using Lakeflow Connect, and then a declarative pipeline performs some transformations on this data to load it into a streaming table.

The declarative pipeline needs to capture the changes made to the streaming table generated by the Lakeflow Connect pipeline. So, how can I enable the `delta.enable.ChangeDataFeed` property on the table generated by Lakeflow Connect? Is there a way to do this? Or do these tables already contain the necessary CDC metadata by default?

Thanks in advance for your answers.

stbjelcevic · 3 weeks ago

Hi @Nmtc9to5 ,

You likely don't need to manually enable delta.enableChangeDataFeed on Lakeflow Connect tables for most use cases, as they typically contain the CDC metadata by default. (docs)

You can check if it is enabled on the table with SHOW TBLPROPERTIES catalog.schema.your_table;

Use STREAM() in your downstream declarative pipeline to incrementally read from the Lakeflow Connect table.

Satyam4u · 3 weeks ago

The short answer is no, you don't need to enable it, because the tables generated by Lakeflow Connect already capture and stream CDC metadata natively. In fact, streaming tables in Delta Live Tables (DLT) and Lakeflow pipelines are built on top of an append-only transaction log mechanism that naturally tracks incremental row changes (inserts, updates, deletes) without needing standard CDF turned on.

drishti_chadha · 3 weeks ago

Lakeflow Connect tables generally already include the CDC metadata required for downstream processing. I'd first check:

SHOW TBLPROPERTIES <catalog>.<schema>.<table>;

If delta.enableChangeDataFeed=true is present, you can directly consume the table with Auto CDC. If not, you may be able to enable CDF via ALTER TABLE, though it's worth confirming whether Lakeflow Connect manages and overrides table properties

Databricks Partner