topic Re: GCP Databricks GKE cluster with 4 nodes in Administration & Architecture

GCP Databricks GKE cluster with 4 nodes

manojpatil04 — Wed, 04 Dec 2024 04:21:46 GMT

I am working on setting up GCP Databricks and successfully created first GCP-Databricks workspace, but what I observed is it is incurring additional charges even i am using 14days free trail. It is GKE cluster with 4 nodes which are spin up as part of workspace creation. Is there way to delete/reduced pods so that additional GCP costs can be reduced.

Re: GCP Databricks GKE cluster with 4 nodes

Louis_Frolio — Wed, 04 Dec 2024 12:28:32 GMT

Databricks on GCP utilizes Google Kubernetes Engine (GKE) clusters as part of its infrastructure. These GKE clusters can incur costs even when no Databricks clusters are actively running. The GKE cluster is used to manage the Databricks environment, and charges apply for the underlying infrastructure

Hope this helps.

Re: GCP Databricks GKE cluster with 4 nodes

manojpatil04 — Thu, 05 Dec 2024 05:59:40 GMT

Thank you @Louis_Frolio,

Just want to dig more into this as is there any way to reduce this nodes using CLI or creating customer managed network.

Re: GCP Databricks GKE cluster with 4 nodes

Louis_Frolio — Fri, 06 Dec 2024 13:57:38 GMT

There are options:

1. Reduce Node Pool Size: e.g. using gcloud clusters resize

2. Enable autoscaling: enable min and max nodes. Use "optimize-utilization" autoscaling profile for more aggressive scaling down.

3. Scale down during idle periods.

4. Consider GKE autopilot.

Hope this helps.