Databricks clusters stuck on Pending and Terminating state indefinitely

[antoooks]

Hi everyone,

Our company is using Databricks on GKE. It works fine until suddenly when we try to create and terminate clusters today, it got stuck on Pending and Terminating state for hours (now more than 6 hours). There is no conclusion can be drawn from Databricks clusters API and Kubernetes, except that it gives "Finding instances for new nodes, acquiring more instances if necessary". I tried restarting all Databricks pods on Kubernetes but it's still happening. Can someone help us about this? Please find screenshot below for more context.

Regards,

Kurnianto

 

[Kaniz]

Hi @antoooks! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[antoooks · Accepted solution]

We have figured it out. So we have to allow 0.0.0.0:443 to our Firewall rules, it's not the best outcome out there since we are using private GKE clusters. However it solves our issue.

Thanks.

[jose_gonzalez]

Hi @Kurnianto Trilaksono Sutjipto​ ,

Thanks for the follow-up. Could you mark your solution as best, so others can quickly find it?

[Mostafa]

Hi everyone!

I have similar issues and I don't know what I should do. please help me out. I am a student and I need this to be fixed to move forward.

It has been a few hours since DB tries to delete those clusters. When I create a new one, it gets stuck in the pending mode forever!

Thank you so much in advance! I am using the community edition.

[gazzyjuruj]

same, it's happening today since morning.

[Databricks_Buil]

Hi @Kurnianto Trilaksono Sutjipto​ : Figured out after multiple connects that This is typically a cloud provider issue. You can file a support ticket if the issue persists.