cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Job Concurrency Queue not working as expected

arielmoraes
New Contributor III

I have a process that should run the same notebook with varying parameters, thus translating to a job with queue and concurrency enabled. When the first executions are triggered the Jobs Runs work as expected, i.e. if the job has a max concurrency set to 10, all executions start running concurrently as expected and new executions are queued up.

arielmoraes_0-1696872175101.png

So, after the current concurrent executions end it's expected the queued items to start being processed as the slots are being freed up, but what is happening is only a few queued items execute concurrently after the first ones finish:

arielmoraes_1-1696872724206.png

Most of the time only a couple of jobs run concurrently after the first batch. 

It appears to be a bug in the queueing mechanism or am I missing something?

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

arielmoraes
New Contributor III

Hi @Retired_mod, we double-checked everything, the resources are enough and all settings are properly set. I'll reach out the support by filing a new ticket. Thank you for your help.

View solution in original post

1 REPLY 1

arielmoraes
New Contributor III

Hi @Retired_mod, we double-checked everything, the resources are enough and all settings are properly set. I'll reach out the support by filing a new ticket. Thank you for your help.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group