We need to set up a job that consists of several hundred tasks with many dependencies between each other. We are considering two different directions:

1. Databricks job with tasks, with dependencies defined as code and deployed with Databricks asset bundles. The resulting job would probably be too complicated to visualise in jobs GUI. However, it is possible to break it down to subjobs, and then a main job orchestrating between the subjobs.

2. One single job, where a notebook does the orchestration, evaluates which task's dependencies are met and then runs it by calling another notebook or by giving parameters to a second, executor task whose job would be to run the notebook. Dependencies are stored in a table that is queried by the orchestrator notebook. This task would run parallelly in multiple threads to make sure that not only one task is run at the same time. Parallelism can be created in Databricks job or in Python.

With option 1, we have the following questions:

• Is there a maximum number of dependencies that a task can get and that jobs can then handle without performance overhead?
• If there is a certain task whose dependencies are not met, does this waiting reserve capacity that could be used elsewhere if this waiting did not need to happen? Or does the orchestration reallocate resources intelligently, and instead of waiting, it starts another task whose dependencies are already met? (Compare it with option 2 where the task selected by the orchestrator notebook is always a task that can be run instantly so there is no waiting.)
• More generally, how many threads is a single job run on? For example, if there are 100 tasks without dependencies in a job, how many of them are started at the same time and how does it depend on the cluster it runs on and the number of workers?

With option 2, we have the following questions:

• If the parallelism is defined in a foreach task, is it possible to implement a do... until logic? We do not know upfront how many times does the orchestrator needs to run. If the answer is no, then the loop needs to be also in Python.
• In option 2, there is only one job running with one or two tasks. Some (or most) of the Databricks logging and monitoring function that goes with jobs is lost, and it has to be programmatically created. How big price is this to pay?

Additionally to the above questions, any general advice in the topic is also welcome!