I'm using Databricks on AWS. Our clusters are typically in PENDING state for 5-8 minutes after they are created. I would like to find out why (ec2 instance provisioning? docker image download is slow? ...?). The cluster logs are not helpful enough because I only see the timestamps of init script execution which is in our case is ~2 seconds.

I'd like to improve startup times. How can I find out what takes so much time to launch the cluster? Is there some logging or event emitting that I can read and analyze?