https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11772#M6697 <P>hi @Sergey Ivanychev​&nbsp;while the cluster is starting, you can see the status on the compute page. Hover the mouse pointer to the green rotating circle on the left of the cluster name. It will give a notification of what is happening on the cluster. While I captured the screenshot, the cluster was finding for new nodes. This might help you to get some insights on why there is a delay.</P><P></P><P><span class="lia-inline-image-display-wrapper" image-alt="image"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1685i942CD2046764B1B6/image-size/large?v=v2&amp;px=999" role="button" title="image" alt="image" /></span></P> Tue, 02 Aug 2022 11:15:58 GMT Prabakar 2022-08-02T11:15:58Z https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11769#M6694 <P>I'm using Databricks on AWS. Our clusters are typically in PENDING state for 5-8 minutes after they are created. I would like to find out why (ec2 instance provisioning? docker image download is slow? ...?). The cluster logs are not helpful enough because I only see the timestamps of init script execution which is in our case is ~2 seconds. </P><P></P><P>I'd like to improve startup times. How can I find out what takes so much time to launch the cluster? Is there some logging or event emitting that I can read and analyze?</P> Mon, 01 Aug 2022 14:15:31 GMT https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11769#M6694 ivanychev 2022-08-01T14:15:31Z https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11770#M6695 <P>Hi @Sergey Ivanychev​&nbsp;, Did you check if there are any legacy init-scripts loaded on the DBFS? Check for any scripts under dbfs:/databricks/init.</P> Mon, 01 Aug 2022 22:51:09 GMT https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11770#M6695 Vivian_Wilfred 2022-08-01T22:51:09Z https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11771#M6696 <P>Nope, there's nothing</P> Tue, 02 Aug 2022 08:34:26 GMT https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11771#M6696 ivanychev 2022-08-02T08:34:26Z https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11772#M6697 <P>hi @Sergey Ivanychev​&nbsp;while the cluster is starting, you can see the status on the compute page. Hover the mouse pointer to the green rotating circle on the left of the cluster name. It will give a notification of what is happening on the cluster. While I captured the screenshot, the cluster was finding for new nodes. This might help you to get some insights on why there is a delay.</P><P></P><P><span class="lia-inline-image-display-wrapper" image-alt="image"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1685i942CD2046764B1B6/image-size/large?v=v2&amp;px=999" role="button" title="image" alt="image" /></span></P> Tue, 02 Aug 2022 11:15:58 GMT https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11772#M6697 Prabakar 2022-08-02T11:15:58Z https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11773#M6698 <P>That sounds promising, thanks! Is this data available via any API or logged anywhere?</P> Tue, 02 Aug 2022 12:06:30 GMT https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11773#M6698 ivanychev 2022-08-02T12:06:30Z https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11774#M6699 <P>Unfortunately, this is not available either on the UI or via API. We get this information only from the backend logs. If you feel this information is required for some analysis, I would recommend raising a feature request for this via our ideas portal. </P><P><A href="https://docs.databricks.com/resources/ideas.html" target="test_blank">https://docs.databricks.com/resources/ideas.html</A></P> Tue, 02 Aug 2022 13:27:38 GMT https://community.databricks.com/t5/data-engineering/how-to-find-out-why-the-cluster-is-in-pending-state-for-so-long/m-p/11774#M6699 Prabakar 2022-08-02T13:27:38Z