https://community.databricks.com/t5/data-engineering/parallelize-spark-jobs-on-the-same-cluster/m-p/15583#M9891 <P>In the past I used direct multi-threaded orchestration inside of driver applications, but that was prior to Databricks supporting multi-task jobs.</P><P></P><P>If you create a job through the workflows tab, you can set up multiple notebooks, python, or jar tasks to run in parallel as well as configure a dependency graph between them if desired.</P><P></P><P>You can either orchestrate those jobs via separate clusters in a single job or share the resources of one or more clusters across different tasks.</P><P></P><P>You can do this without having to assign a schedule to the job.</P><P>​</P><P></P><P>Hope this helps!</P> Wed, 29 Jun 2022 18:45:45 GMT ron_defreitas 2022-06-29T18:45:45Z https://community.databricks.com/t5/data-engineering/parallelize-spark-jobs-on-the-same-cluster/m-p/15582#M9890 <P>Whats the best way to parallelize multiple spark jobs on the same cluster during a backfill?</P> Wed, 29 Jun 2022 16:51:16 GMT https://community.databricks.com/t5/data-engineering/parallelize-spark-jobs-on-the-same-cluster/m-p/15582#M9890 OliverLewis 2022-06-29T16:51:16Z https://community.databricks.com/t5/data-engineering/parallelize-spark-jobs-on-the-same-cluster/m-p/15583#M9891 <P>In the past I used direct multi-threaded orchestration inside of driver applications, but that was prior to Databricks supporting multi-task jobs.</P><P></P><P>If you create a job through the workflows tab, you can set up multiple notebooks, python, or jar tasks to run in parallel as well as configure a dependency graph between them if desired.</P><P></P><P>You can either orchestrate those jobs via separate clusters in a single job or share the resources of one or more clusters across different tasks.</P><P></P><P>You can do this without having to assign a schedule to the job.</P><P>​</P><P></P><P>Hope this helps!</P> Wed, 29 Jun 2022 18:45:45 GMT https://community.databricks.com/t5/data-engineering/parallelize-spark-jobs-on-the-same-cluster/m-p/15583#M9891 ron_defreitas 2022-06-29T18:45:45Z https://community.databricks.com/t5/data-engineering/parallelize-spark-jobs-on-the-same-cluster/m-p/15585#M9893 <P>Hi @Oliver Lewis​,</P><P></P><P>Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.</P> Tue, 05 Jul 2022 17:30:10 GMT https://community.databricks.com/t5/data-engineering/parallelize-spark-jobs-on-the-same-cluster/m-p/15585#M9893 jose_gonzalez 2022-07-05T17:30:10Z