topic Re: Does Databricks need GCP VMs for a workspace with no clusters in it? in Data Engineering

Does Databricks need GCP VMs for a workspace with no clusters in it?

inpefess — Mon, 05 Sep 2022 16:16:18 GMT

Hi! I'm using GCP. Does Databricks workspace always need two e2-highmem-2 instances running as soon as I create a workspace? I seem them in my VM list in GCP console no matter what (I can stop or remove a cluster, but these two machines are always there). Is there a way to shut them down automatically when I'm not using the workspace?

Re: Does Databricks need GCP VMs for a workspace with no clusters in it?

Debayan — Mon, 05 Sep 2022 20:55:32 GMT

Hi @Boris Shminke , Thanks for reaching out to community.databricks.com.

When you create a workspace in databricks account console, in the background it creates a cluster with small nodes to carry out workspace operations. As of now, there is no option to pause the cluster nodes . The only option is to delete the workspace when it is not used. Even if you delete the nodes from the backend GKE cluster, Databricks service will automatically create it for recovery.

I would request to file a feature request in our ideas portal https://ideas.databricks.com/ideas/, so that you can interact and receive the updates directly with PM team on this idea. The ideas portal allows the customers to submit enhancement ideas and other customers to vote on their submitted ideas, and also gives the customers a direct line of communication with PM. You can subscribe to status updates as well.

Re: Does Databricks need GCP VMs for a workspace with no clusters in it?

abagshaw — Tue, 27 Jun 2023 18:44:14 GMT

To clarify, on Databricks on GCP will automatically delete the underlying GKE after 5 days of inactivity (no cluster launches or non-empty instance pools) in the workspace. You can contact Databricks support if you want to shorten the idle TTL for the GKE associated with your workspace.

Re: Does Databricks need GCP VMs for a workspace with no clusters in it?

marce3223 — Wed, 08 Nov 2023 17:00:35 GMT

Can you explain why GCP needs these 2 instances and AWS and Azure do not? Where is this documented/explained in the Databricks documentation?

Thanks

Re: Does Databricks need GCP VMs for a workspace with no clusters in it?

marce3223 — Wed, 08 Nov 2023 17:03:14 GMT

Do you have the references to the documentation that explains all these things?