amirabedhiafi
Contributor

Hello @shan-databricks  !

One additional point, I would also validate the expected load with the SQL Server DBA because even if Lakeflow manages the parallelism internally the source SQL Server still needs to handle those concurrent reads. For 100 tables, I would start with one pipeline/gateway, monitor extraction duration and SQL Server load, then only split into multiple pipelines/gateways if there is a clear operational need such as different refresh SLAs, very large tables, or source side throttling. Don't forget that for tables with primary keys CT is generally preferred over CDC to reduce source overhead.

If this answer resolves your question, could you please mark it as “Accept as Solution”? It will help other users quickly find the correct fix.

Senior BI/Data Engineer | Microsoft MVP Data Platform | Microsoft MVP Power BI | Power BI Super User | C# Corner MVP