@AchintyaSingh 

Databricks does not support clusters with multiple drivers to run new jobs in parallel. Each Databricks cluster has a single driver node, allowing only one job at a time.
Workarounds for Achieving Parallel Job Execution:

1. Multiple Clusters:
- Create Multiple Job Clusters: Set up multiple clusters, each with its own driver node, to run different jobs in parallel. This lets you submit different Spark applications with varied arguments to separate clusters.
- Autoscaling Support: Configure these clusters with autoscaling to efficiently manage the workload. You can set a range for the number of workers, allowing Databricks to dynamically adjust resources based on job requirements.
2. Job Scheduling and Orchestration:
- Databricks Workflows: Utilize Databricks Workflows for scheduling and orchestrating multiple jobs. Define tasks with dependencies and run them in parallel where applicable.
- External Orchestration Tools: Use tools like Apache Airflow or Azure Data Factory to manage and run multiple Databricks jobs in parallel.