Tech Companies & Lakeflow Connect

[anushnagesh]

Hello fellow techies,

For the past few days I have been doing some R&D wrt Lakeflow connect. I came to the understanding that, Lakeflow Ingestion Pipeline and SQL server connection have a 1:1 mapping. This would mean that as number of servers scales number of pipelines will grow. 
There are pros and cons to this. Pros being, delta ingestion is managed, cost is lesser and so on whereas cons being too many pipeline to monitor and fix if issues arise, and just the sheer volume of pipelines might intimidate people.
We can use YAML and use DAB to make it configurable and deployable but the fact that we need to monitor soo many pipeline remains.

I am very curious how big tech companies approach this problem? How does an organization decide whether to leverage lakeflow connect or stick to writing a completely custom code for ingestionm?
People who work as data engineers or been a part of developing ingestion pipelines can you please provide insights into this?

Thanking you,
Anush

[Satyasai]

Most modern enterprise data platforms favor managed ingestion (Lakeflow Connect) for standard relational databases like SQL Server. The operational complexity of monitoring 50 managed pipelines is significantly lower than maintaining custom CDC code, offset management, and schema-drift handling for 50 separate connections.

By combining Lakeflow Connect with Databricks Asset Bundles (DABs) for deployment and System Tables for centralized monitoring, teams achieve a scalable, low-maintenance ingestion layer.

This deep-dive video walks through the architecture, setup, and scaling considerations for Lakeflow Connect database ingestion:

https://www.youtube.com/watch?v=sZ3wVoedP64