Hi team,

We created a workflow and attached it to a job cluster (which is configured to use compute pool). When we run the pipeline, it takes up to 5 minutes to go into clusterReady state and this is adding latency to our use case. Even with subsequent runs, it's waiting for cluster to be ready. Can someone please help me understand how to reduce the overall latency and better way of using job compute.

We tried with serverless warehouse (non SQL) and it's adding around 20-25 seconds latency for each task in the job. In screenshot (Screenshot 2025-04-03 ar 3:43:23 PM), the task took 33 seconds but notebook cell has run only for 16 seconds. Would like to understand what is adding up to latency in this case.

Thanks & Regards,

Bhargava