Since Workflows (Multi-Task jobs) is now GA, one way to work around the 1000 concurrent jobs limit is to use tasks within a job. Each job can have 100 tasks, and these tasks do not count toward the concurrent job limit.
Since Workflows (Multi-Task jobs) is now GA, one way to work around the 1000 concurrent jobs limit is to use tasks within a job. Each job can have 100 tasks, and these tasks do not count toward the concurrent job limit.
Welcome to Databricks Community: Lets learn, network and celebrate together
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.