Hi Karthik,

I found your blog very helpful, and it closely matches what I am trying to implement. However, I’m not able to get incremental ingestion working as expected: each time I run a pipeline update, it appears to read all rows from the source instead of only the changes.

My setup is:

• Source: SQL Server database on an Azure VM, with Change Tracking (CT) enabled.
• Databricks: Lakeflow Connect managed ingestion, using a DLT pipeline created via the API. Currently there is no Gateway pipeling setup.

Based on my analysis and the pipeline logs, I have a few questions:

1. When creating the pipeline via API, I used
   "table_configuration": { "cdc_mechanism": "change_tracking" }
   and the creation call succeeded.

   • Is "cdc_mechanism": "change_tracking" the correct field name and value to enable SQL Server Change Tracking for a managed ingestion pipeline?

2. In the UI/metadata, the flow type is shown as SNAPSHOT_CHANGE.

   • Is SNAPSHOT_CHANGE the expected flow type when CT is enabled, or should I expect to see a different type (for example, something explicitly indicating CHANGE_TRACKING)?

3. Regarding release channels:

   • Is CT support for SQL Server in Lakeflow Connect only available on the PREVIEW channel, or should it work on the CURRENT channel as well?

Any guidance on the correct configuration for CT-based incremental ingestion (and how to verify that the connector is actually using CT rather than performing full scans) would be greatly appreciated.

Thanks

Swarup