article Simplifying external data ingestion with Lakeflow Connect in Technical Blog

Simplifying external data ingestion with Lakeflow Connect

Karthik_Ballull — Fri, 15 Nov 2024 10:37:57 GMT

As data engineering and analytics become increasingly complex, organizations often seek to integrate the scalability and flexibility of the cloud with the robustness of traditional relational databases. The Databricks lakehouse, with its recent introduction of the Lakeflow Connect, seamlessly bridges these two worlds, allowing teams to harness the full power of Azure SQL while leveraging Databricks advanced capabilities for big data and AI workloads.

Re: Simplifying external data ingestion with Lakeflow Connect

ddharma — Sun, 08 Jun 2025 04:51:35 GMT

Dear Karthik, thank you so much for such a comprehensive deep dive on Lakeflow connect.

In our scenario, we are looking for Azure Synapse < Connector & Migration > to Databricks.

Did explore the following references, however, don't see the relevant connector or support, as yet:

https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/

https://docs.databricks.com/aws/en/ingestion/

Any ideas / suggestions?

FYI, the lakehouse federation supports Azure Synpase Connector as detailed here.

Thanks & Regards,

Dileep

Re: Simplifying external data ingestion with Lakeflow Connect

ddharma — Sun, 08 Jun 2025 05:30:09 GMT

Dear Karthik,

Was trying to edit my above comment to augment this, however, seems to have timed out.

For any migration, typically, we have 4 options:

1. Leverage Data Platform specific Native Services (in this case Lakeflow)

// most preferred as it's comprehensive & integrates across the data-value-chain like catalog and more

2. Leverage Cloud Native Services like ADF (relevant solution here, thanks @Rishabh-Pandey)

// supports variety of connectors & fully scalable powered by deep cloud integrations

3. Leverage off-the-shelf $paid tools or OSS

// good option, but needs evaluation, (licensing), suitability across business scenarios

4. Develop custom accelerators / custom code / automation scripts or leverage OSS

// surely can be done but preferred (ideally) in the absence of 1-3

Surely Lakehouse Federation is also an option, however, in our case we intend to migrate and decommission Azure Synapse over time (post cut over etc).

Given the first preference is option 1 Lakeflow, was exploring source connector availability to Azure Synapse to help migrate to Databricks Delta Lake.

HTH provide more context on the approach & thought process.

Thanks & regards,

Dileep

Re: Simplifying external data ingestion with Lakeflow Connect

Rishabh-Pandey — Mon, 09 Jun 2025 18:15:02 GMT

Thanks @ddharma for tagging me and i am impressed that you loved my solution.

Re: Simplifying external data ingestion with Lakeflow Connect

ddharma — Wed, 11 Jun 2025 04:16:05 GMT

@Karthik_Ballull and @Rishabh-Pandey any thoughts / comments on my original comments above?

1. Does Lakeflow provide source connector for Azure Synapse?

2. Or if you are aware about any ETA?

Esp. as Unity Catalog has support for Azure Synapse connector.

Thanks & Regards,

Dileep

Re: Simplifying external data ingestion with Lakeflow Connect

Rishabh-Pandey — Wed, 11 Jun 2025 06:03:34 GMT

@ddharma Lakeflow will surely have azure synapse connector but not now , they have in their roadmap.

Re: Simplifying external data ingestion with Lakeflow Connect

Swarup_DE — Wed, 27 May 2026 03:38:52 GMT

Hi Karthik,

I found your blog very helpful, and it closely matches what I am trying to implement. However, I’m not able to get incremental ingestion working as expected: each time I run a pipeline update, it appears to read all rows from the source instead of only the changes.

My setup is:

Source: SQL Server database on an Azure VM, with Change Tracking (CT) enabled.
Databricks: Lakeflow Connect managed ingestion, using a DLT pipeline created via the API. Currently there is no Gateway pipeling setup.

Based on my analysis and the pipeline logs, I have a few questions:

When creating the pipeline via API, I used
"table_configuration": { "cdc_mechanism": "change_tracking" }
and the creation call succeeded.
- Is "cdc_mechanism": "change_tracking" the correct field name and value to enable SQL Server Change Tracking for a managed ingestion pipeline?
In the UI/metadata, the flow type is shown as SNAPSHOT_CHANGE.
- Is SNAPSHOT_CHANGE the expected flow type when CT is enabled, or should I expect to see a different type (for example, something explicitly indicating CHANGE_TRACKING)?
Regarding release channels:
- Is CT support for SQL Server in Lakeflow Connect only available on the PREVIEW channel, or should it work on the CURRENT channel as well?

Any guidance on the correct configuration for CT-based incremental ingestion (and how to verify that the connector is actually using CT rather than performing full scans) would be greatly appreciated.

Thanks

Swarup