TLDR: Do not scale Lakeflow Connect for MySQL linearly by creating dozens of identical pipelines. Current guidance is ~250 tables per pipeline, with a hard limit of 1,000 flows / ~500 tables, and the MySQL connector has been tested at 100 tables with <1 TB snapshot data. The gateway must also run continuously to avoid binlog cleanup/full refresh issues.

Options

1. Tiered Lakeflow Connect sharding

   • Split pipelines by criticality / change rate / table size, not just by table count.
   • Keep hot, business-critical tables in Lakeflow Connect; put cold/rarely used tables on less frequent schedules.
   • Stagger pipeline schedules so ingestion pipelines don’t all run at once, while gateways stay up.

2. Consolidate before ingest

   • If many source tables are structurally similar, consolidate at source or landing into fewer larger destination tables/views.
   • The guidance for very high table-count scenarios is that some consolidation is necessary; otherwise you hit table/platform limits and cost grows quickly.

3. Use external CDC for the long tail

   • For 6,000–7,000 tables / multi-TB estates, use Debezium/Kafka or object storage landing + Auto Loader / SDP for the bulk estate, and reserve Lakeflow Connect for the subset that benefits most from managed CDC.
   • We recommend Debezium -> Auto Loader / SDP as a reasonable architecture for this scale because it allows consolidation and avoids massive pipeline sprawl.

Recommendation

Recommend Option 3 + Option 1 together:
Use Lakeflow Connect only for the top-priority subset of MySQL tables, grouped into a small number of carefully sharded pipelines, and move the long tail to an external CDC landing pattern. That is the most scalable path for thousands of tables / multi-TB while minimizing cluster proliferation, ops overhead, and cost.