Databricks Lakeflow Connect for MySQL

antoalphi — Thu, 09 Apr 2026 06:13:41 GMT

In Databricks Lakeflow Connect for MySQL (currently in public preview), Databricks recommends limiting each ingestion pipeline to around 250 tables, with validated testing up to 1 TB of snapshot data.

However, in real-world enterprise scenarios, customers often have significantly larger environments for example, thousands of tables (e.g., 6,000–7,000) and data volumes exceeding multiple terabytes.

To accommodate this, we are required to create multiple ingestion pipelines. Since each pipeline typically provisions its own compute resources (clusters), this can lead to:

Increased infrastructure costs due to multiple clusters running in parallel
Higher operational overhead in managing multiple pipelines
Customer dissatisfaction due to perceived inefficiency and cost escalation

This raises an important challenge:
How can we design a scalable ingestion strategy that handles large table volumes and data sizes efficiently, while minimizing compute cost and avoiding unnecessary cluster proliferation?

Re: Databricks Lakeflow Connect for MySQL

Sumit_7 — Thu, 09 Apr 2026 06:34:51 GMT

@antoalphi I think you have already answered in the first line itself - Public Preview -- meaning not full developed for general/real use. Hence it comes with limitations or bugs which will be covered in General Available. Though still you may consider following points to tackle effectively:

Group tables - split pipelines by schema / domain / size (not randomly)
Use incremental (CDC) instead of full snapshot - reduces compute drastically
Orchestrate pipelines (Databricks Workflows) - run sequentially or staggered to avoid many clusters at once
Use serverless pipelines (where supported) - reduces cluster management overhead

Hope this helps, thanks.

topic Databricks Lakeflow Connect for MySQL in Community Articles

Databricks Lakeflow Connect for MySQL

Re: Databricks Lakeflow Connect for MySQL