Databricks serverless Queue based on Serverless Environment

[harshgrewal27]

So for my Databricks Workflows , i'd a job that was using Environment 3 of Serverless Compute with Performace Optimized as enabled , as we wanted quick execution of Job when triggered . There can be around 10-20 concurrent run , but noticed maybe of the job run stayed in queue even after other runs got completed . 

Also when i switched to Env 4 , so same job had no queue time , but had increased execution time .

So is there any documentation about how does different environment handle queuing , and concurrent runs?

[CURIOUS_DE]

Hey @harshgrewal27 

Before we start there a concept called“container reuse window” that often explains why some jobs start instantly and others queue."

1. Queueing is not controlled by Environment version (Env 3 / Env 4).
   Queueing in Databricks Workflows mainly happens due to:
   1. Job max concurrent runs
   2. Workspace concurrency limits
   3. Serverless compute capacity scheduling( Optional as it omits most execution because of no custom capacity orchestration for Prod Jobs)
      

If many runs start at once (10–20 or may be longer 30 min), some runs may enter the scheduler queue until compute becomes available.

2. Why Env 3 showed queue but Env 4 didn’t

Most likely behavior:

• Env 3 + Performance Optimized
  • Jobs execute faster
  • Each run requests higher compute resources(32GB-Max for High Compute)
  • Serverless pool may temporarily run out of slots ( Experienced a lot so pointing this out)
    → Some runs wait in queue

• Env 4

  • Scheduler provisions compute more gradually

  • Runs start immediately
  • But execution time is slightly longer(Because of computer scheduling and reuse window)

3. What Databricks documentation says

Docs mention that job runs queue when concurrency or compute capacity limits are reached, not based on environment version.

Relevant areas in documentation:

• Job concurrency settings

• Serverless compute capacity allocation

Link:- https://docs.databricks.com/aws/en/release-notes/serverless/environment-version?

IMP Point:- concurrent run limits or scheduling behavior. There’s no explicit note that Env‑3 can “queue” more conservatively than Env‑4 or vice versa (the differences are mostly library/runtime features) 

[harshgrewal27]

For concurrent Run i kept it as 50 , but still even before reaching them , they started queuing . Also kept 16GB serverless memory for each environment.

But can check Serverless pool to increase its capacity and improve performance . Thanks for you valuable insights

[aleksandra_ch]

Hi @harshgrewal27 ,

Did you compare query profiles of a run in Env 3 vs Env 4? Check for:

• Amount of data being processed per run
• Longest running stages

This might also explain the execution time difference between Env 3 and Env 4.

Best regards,