cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Does Databricks need GCP VMs for a workspace with no clusters in it?

inpefess
New Contributor II

Hi! I'm using GCP. Does Databricks workspace always need two e2-highmem-2 instances running as soon as I create a workspace? I seem them in my VM list in GCP console no matter what (I can stop or remove a cluster, but these two machines are always there). Is there a way to shut them down automatically when I'm not using the workspace?

4 REPLIES 4

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi @Boris Shminkeโ€‹ , Thanks for reaching out to community.databricks.com.

When you create a workspace in databricks account console, in the background it creates a cluster with small nodes to carry out workspace operations. As of now, there is no option to pause the cluster nodes . The only option is to delete the workspace when it is not used. Even if you delete the nodes from the backend GKE cluster, Databricks service will automatically create it for recovery.

I would request to file a feature request in our ideas portal https://ideas.databricks.com/ideas/, so that you can interact and receive the updates directly with PM team on this idea. The ideas portal allows the customers to submit enhancement ideas and other customers to vote on their submitted ideas, and also gives the customers a direct line of communication with PM. You can subscribe to status updates as well.

marce3223
New Contributor II

Can you explain why GCP needs these 2 instances and AWS and Azure do not? Where is this documented/explained in the Databricks documentation?

Thanks

abagshaw
New Contributor III

To clarify, on Databricks on GCP will automatically delete the underlying GKE after 5 days of inactivity (no cluster launches or non-empty instance pools) in the workspace. You can contact Databricks support if you want to shorten the idle TTL for the GKE associated with your workspace. 

marce3223
New Contributor II

Do you have the references to the documentation that explains all these things?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group