https://community.databricks.com/t5/get-started-discussions/running-jar-jobs-in-parallel-on-a-cluster/m-p/72058#M3091 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/106264">@AchintyaSingh</a>&nbsp;</P> <P>Databricks does not support clusters with multiple drivers to run new jobs in parallel. Each Databricks cluster has a single driver node, allowing only one job at a time.<BR />Workarounds for Achieving Parallel Job Execution:</P> <P>1. Multiple Clusters:<BR />- Create Multiple Job Clusters: Set up multiple clusters, each with its own driver node, to run different jobs in parallel. This lets you submit different Spark applications with varied arguments to separate clusters.<BR />- Autoscaling Support: Configure these clusters with autoscaling to efficiently manage the workload. You can set a range for the number of workers, allowing Databricks to dynamically adjust resources based on job requirements.<BR />2. Job Scheduling and Orchestration:<BR />- Databricks Workflows: Utilize Databricks Workflows for scheduling and orchestrating multiple jobs. Define tasks with dependencies and run them in parallel where applicable.<BR />- External Orchestration Tools: Use tools like Apache Airflow or Azure Data Factory to manage and run multiple Databricks jobs in parallel.</P> Fri, 07 Jun 2024 14:23:28 GMT Yeshwanth 2024-06-07T14:23:28Z https://community.databricks.com/t5/get-started-discussions/running-jar-jobs-in-parallel-on-a-cluster/m-p/71722#M3080 <P>Hi everyone, I'm trying to find out if databricks has support for clusters which can scale out with more drivers to run new jobs in parallel. If not, then is there a work around for this? I've noticed that all-purpose and job compute clusters both feature only a single driver.</P><P>I'm trying to run my spark applications from a jar file passing different arguments to it on every run. I need the applications to be run in parallel and not sequentially or concurrently, this is because I have a pretty strict time constraint requirement. I also need auto-scaling support for the same reason.</P><P>I'm quite new to databricks and spark as well, would greatly appreciate anyone's input.</P> Wed, 05 Jun 2024 09:34:59 GMT https://community.databricks.com/t5/get-started-discussions/running-jar-jobs-in-parallel-on-a-cluster/m-p/71722#M3080 AchintyaSingh 2024-06-05T09:34:59Z https://community.databricks.com/t5/get-started-discussions/running-jar-jobs-in-parallel-on-a-cluster/m-p/72058#M3091 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/106264">@AchintyaSingh</a>&nbsp;</P> <P>Databricks does not support clusters with multiple drivers to run new jobs in parallel. Each Databricks cluster has a single driver node, allowing only one job at a time.<BR />Workarounds for Achieving Parallel Job Execution:</P> <P>1. Multiple Clusters:<BR />- Create Multiple Job Clusters: Set up multiple clusters, each with its own driver node, to run different jobs in parallel. This lets you submit different Spark applications with varied arguments to separate clusters.<BR />- Autoscaling Support: Configure these clusters with autoscaling to efficiently manage the workload. You can set a range for the number of workers, allowing Databricks to dynamically adjust resources based on job requirements.<BR />2. Job Scheduling and Orchestration:<BR />- Databricks Workflows: Utilize Databricks Workflows for scheduling and orchestrating multiple jobs. Define tasks with dependencies and run them in parallel where applicable.<BR />- External Orchestration Tools: Use tools like Apache Airflow or Azure Data Factory to manage and run multiple Databricks jobs in parallel.</P> Fri, 07 Jun 2024 14:23:28 GMT https://community.databricks.com/t5/get-started-discussions/running-jar-jobs-in-parallel-on-a-cluster/m-p/72058#M3091 Yeshwanth 2024-06-07T14:23:28Z