cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Does Databricks need GCP VMs for a workspace with no clusters in it?

inpefess
New Contributor II

Hi! I'm using GCP. Does Databricks workspace always need two e2-highmem-2 instances running as soon as I create a workspace? I seem them in my VM list in GCP console no matter what (I can stop or remove a cluster, but these two machines are always there). Is there a way to shut them down automatically when I'm not using the workspace?

4 REPLIES 4

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi @Boris Shminke​ , Thanks for reaching out to community.databricks.com.

When you create a workspace in databricks account console, in the background it creates a cluster with small nodes to carry out workspace operations. As of now, there is no option to pause the cluster nodes . The only option is to delete the workspace when it is not used. Even if you delete the nodes from the backend GKE cluster, Databricks service will automatically create it for recovery.

I would request to file a feature request in our ideas portal https://ideas.databricks.com/ideas/, so that you can interact and receive the updates directly with PM team on this idea. The ideas portal allows the customers to submit enhancement ideas and other customers to vote on their submitted ideas, and also gives the customers a direct line of communication with PM. You can subscribe to status updates as well.

marce3223
New Contributor II

Can you explain why GCP needs these 2 instances and AWS and Azure do not? Where is this documented/explained in the Databricks documentation?

Thanks

abagshaw
New Contributor III

To clarify, on Databricks on GCP will automatically delete the underlying GKE after 5 days of inactivity (no cluster launches or non-empty instance pools) in the workspace. You can contact Databricks support if you want to shorten the idle TTL for the GKE associated with your workspace. 

marce3223
New Contributor II

Do you have the references to the documentation that explains all these things?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.