GCP Databricks GKE cluster with 4 nodes

[manojpatil04]

I am working on setting up GCP Databricks and successfully created first GCP-Databricks workspace, but what I observed is it is incurring additional charges even i am using 14days free trail. It is GKE cluster with 4 nodes which are spin up as part of workspace creation. Is there way to delete/reduced pods so that additional GCP costs can be reduced.

[Louis_Frolio]

Databricks on GCP utilizes Google Kubernetes Engine (GKE) clusters as part of its infrastructure. These GKE clusters can incur costs even when no Databricks clusters are actively running. The GKE cluster is used to manage the Databricks environment, and charges apply for the underlying infrastructure

Hope this helps.

[manojpatil04]

Thank you @Louis_Frolio,

Just want to dig more into this as is there any way to reduce this nodes using CLI or creating customer managed network.

[Louis_Frolio]

There are options:

1. Reduce Node Pool Size: e.g. using gcloud clusters resize

2. Enable autoscaling: enable min and max nodes. Use "optimize-utilization" autoscaling profile for more aggressive scaling down.

3. Scale down during idle periods.

4. Consider GKE autopilot.

Hope this helps.