From Fragmented Schedulers to Unified Orchestration: A Lakehouse Evolution

wesleyfelipe — Mon, 18 May 2026 15:13:11 GMT

This article wraps up a technical deep dive into building large-scale Lakehouse architectures, revisiting design decisions from a 2019 platform that processed billions of payment records.

In the original platform, streaming pipelines ran on Spark Streaming while batch workflows were managed by Control-M. The split worked, but created a fragmented operational model with no unified view of pipelines or dependencies. Grouping jobs onto shared clusters reduced costs but introduced new trade-offs: loss of isolation, application-level retry logic, and reduced observability.

A real chargeback processing scenario illustrates how these challenges compounded — streaming jobs, batch file arrivals, and external partner dependencies all had to be monitored across independent systems, making root-cause analysis slow.

Modern capabilities like Lakeflow show how orchestration can be absorbed into the platform itself: unified batch and streaming, platform-native retries, task-level isolation, and integrated observability.

Key takeaways:

Unified orchestration reduces system fragmentation
Platform-native retries beat ad-hoc application logic
End-to-end observability shortens recovery time
Task-level isolation prevents shared-cluster failure cascades

This part also closes the series, bringing ingestion, SCD processing, governance, and orchestration together into a single view of a modern Lakehouse with ZeroBus, Lakeflow, and Unity Catalog.

🔗 Full article: https://medium.com/@wesley.felipe/databricks-lakehouse-without-the-workarounds-part-4-orchestration-and-pipeline-reliability-84abc7720640

topic From Fragmented Schedulers to Unified Orchestration: A Lakehouse Evolution in Community Articles

From Fragmented Schedulers to Unified Orchestration: A Lakehouse Evolution