Lakeflow Connect: Data Ingestion from SQL Server to Databricks

[shan-databricks]

We have a use case to ingest data from SQL Server into Databricks using Lakeflow Connect. There are 100 tables, and on a daily basis we will perform inserts, updates, and deletes based on CDC data. For this requirement, how can we enable multiple parallel connections to the SQL Server database?

[Ashwin_DSA]

Hi @shan-databricks,

Databricks recommends up to ~250 tables per pipeline, so 100 is well within guidance. Lakeflow Connect doesn’t offer a user-facing control for multiple parallel connections. Instead, configure a single SQL Server gateway with sufficient cores. Databricks automatically manages the parallel JDBC connections from the gateway to your SQL Server.

When you give it enough cores (via the gateway’s compute policy/node sizes) to let Databricks scale extraction in parallel... the platform then opens and manages multiple JDBC connections internally. 

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

[amirabedhiafi]

Hello @shan-databricks  !

One additional point, I would also validate the expected load with the SQL Server DBA because even if Lakeflow manages the parallelism internally the source SQL Server still needs to handle those concurrent reads. For 100 tables, I would start with one pipeline/gateway, monitor extraction duration and SQL Server load, then only split into multiple pipelines/gateways if there is a clear operational need such as different refresh SLAs, very large tables, or source side throttling. Don't forget that for tables with primary keys CT is generally preferred over CDC to reduce source overhead.