GCP Databricks GKE cluster with 4 nodes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-03-2024 08:21 PM
I am working on setting up GCP Databricks and successfully created first GCP-Databricks workspace, but what I observed is it is incurring additional charges even i am using 14days free trail. It is GKE cluster with 4 nodes which are spin up as part of workspace creation. Is there way to delete/reduced pods so that additional GCP costs can be reduced.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2024 04:28 AM
Databricks on GCP utilizes Google Kubernetes Engine (GKE) clusters as part of its infrastructure. These GKE clusters can incur costs even when no Databricks clusters are actively running. The GKE cluster is used to manage the Databricks environment, and charges apply for the underlying infrastructure
Hope this helps.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-04-2024 09:59 PM
Thank you @Louis_Frolio,
Just want to dig more into this as is there any way to reduce this nodes using CLI or creating customer managed network.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2024 05:57 AM
There are options:
1. Reduce Node Pool Size: e.g. using gcloud clusters resize
2. Enable autoscaling: enable min and max nodes. Use "optimize-utilization" autoscaling profile for more aggressive scaling down.
3. Scale down during idle periods.
4. Consider GKE autopilot.
Hope this helps.
