topic Re: Databricks doesn't stop compute resources in GCP in Data Engineering

Databricks doesn't stop compute resources in GCP

jose_herazo — Mon, 16 Jan 2023 19:59:51 GMT

I started using Databricks in Google Cloud but it charges some unexpected costs.

When I create a cluster I notice some compute resources being created in GCP but when I stop the cluster these resources are still up and never shut down. This issue results in some additional charges that exceed the cost per DBU used.

I noticed some alerts in Kubernets that say:

Pod is blocking scale down because it doesn’t have enough Pod Disruption Budget (PDB)
Can’t scale up a node pool because of a failing scheduling predicate

I'm not sure if it is related to this issue in any way.

Thanks!

Re: Databricks doesn't stop compute resources in GCP

-werners- — Tue, 17 Jan 2023 10:38:15 GMT

how do you deploy your cluster(s)?

I see you mentioning k8s, so it might be the config.

But it could also be a Databricks bug as dbrx is only available on GCP for a short while.

Re: Databricks doesn't stop compute resources in GCP

jose_herazo — Tue, 17 Jan 2023 17:52:42 GMT

Basically I just created some singlenode clusters inside Databricks, without configuring anything in GCP.

I see after cluster creation Databricks creates some resources in GCP (Kubernetes clusters, compute instances and instance groups). I deleted these singlenode clusters in databricks but resources in GCP keep active. I tried to delete them directly in GCP, and it works, but minutes later they are automatically created again.

I have been using Databricks on Azure without any issues in billing. I also think it could be something related to GCP.

Re: Databricks doesn't stop compute resources in GCP

-werners- — Wed, 18 Jan 2023 06:47:01 GMT

Strange.

That does seem like a bug/feature indeed; or there is something else running on dbrx like a sql warehouse or a job?

Re: Databricks doesn't stop compute resources in GCP

jose_herazo — Wed, 18 Jan 2023 16:13:49 GMT

Not actually, I just have a blank workspace. I had had created a workspace with a small table in the hivemetastore, one singlenode cluster and a job, but I completely deleted that workspace.

Re: Databricks doesn't stop compute resources in GCP

antquinonez — Mon, 13 Feb 2023 22:34:12 GMT

The answer to the question about the kubernetes cluster regardless of dbx compute and dwh resources running is provided in this thread: https://community.databricks.com/s/question/0D58Y00009TbWqtSAF/auto-termination-for-clusters-jobs-and-delta-live-tables-does-not-terminate-clusters-on-gcp