Hey @bhargavabasava ,
Job Cluster + Compute Pools: Long Startup Times
If you’re using Job Clusters backed by compute pools, the initial delay (~5 minutes) is usually due to cluster provisioning. While compute pools are designed to reduce cold start times by pre-warming VMs, startup latency can still occur if:
There are no idle VMs available in the pool (e.g., 0 clusters in idle state).
The cluster needs to install libraries or run init scripts, which adds to the boot time.
Serverless Jobs Latency (~20–25 seconds overhead)
The behavior you’re seeing where the notebook logic takes 16 seconds but the task duration is 33 seconds is expected when using Serverless compute for Jobs (non-SQL). There is a small but consistent overhead due to orchestration, environment setup, and logging.
That said, serverless jobs generally start much faster than job clusters and offer more predictable latency, so a 20–25 second overhead is considered normal.
Suggestions to Reduce Latency
Use instance pools with Idle Instance Auto Termination set to ~10 minutes. This allows reusing VMs across runs without incurring full provisioning times.
If you’re using isolated job clusters, try to chain multiple tasks in a single job using dependencies. This way, only the first task pays the cold-start penalty, and the following tasks run on the same cluster.
Hope this helps 🙂
Isi
