At Databricks, more and more teams are starting to use Lakeflow (Jobs & Pipelines), and some companies use it primarily for orchestrating scripts in Databricks. It's a powerful tool for building dependencies, but as the number of jobs grows, problems arise due to the lack of convenient monitoring.
When you open Jobs & Pipelines, you see a list of jobs, but you don't see how they interact with each other or how dense they are.
There are dozens of tasks, most of which are scheduled for around 8 a.m. The result is chaos that's hard to see but has a significant impact on the infrastructure. Sometimes you might run out of IP addresses in your subnet or the number of reserved resources. And sometimes you just want to choose a free window, but how? In this article, I'll offer my solution.
