KoenZandvliet
Databricks Partner

Unfortunately, running multiple jobs in parallel using a single job cluster is not supported (yet). New in databricks is the possibility to create a job that orchestrates multiple jobs. These jobs will however still use their own cluster (configuration).

In the case the time of instancing a cluster takes much longer than executing an actual job, you could consider using `all-purpose` cluster. Depending on your use-case it might also be possible to rewrite/re-configure your 5 jobs to a single job with 5 tasks.

 

View solution in original post