Concurrent Workflow Jobs

holychs
Databricks Partner

‎12-05-2024 08:02 AM

Hi Community, 

I am trying to run a Databricks workflow job using run_job_task under a for_loop. I have set the concurrent jobs as 2. 

I can see 2 iteration jobs getting triggered successfully. But both fail with an error:

"ConnectException: Connection refused (Connection refused)

Error while obtaining a new communication channel"
Contradictory to the fact that same job gets completed successfully with concurrency as 1. 
Cluster Configuration: Driver: Standard_DS5_v2 · Workers: Standard_DS5_v2 · 4-8 workers
 
Labels:
0 Kudos