topic Re: Terraform Destroy not able to prune Databricks Provisioned GKE Cluster on GCP in Administration & Architecture

Terraform Destroy not able to prune Databricks Provisioned GKE Cluster on GCP

KLin — Tue, 29 Oct 2024 15:52:42 GMT

Hi there,

newbie here in Databricks on GCP. I provisioned my Databricks workspace with Terraform and all worked well. Now when I would like to target destroy my workspace, issues occur:

When I do terraform destroy -target module.workspace, the workspace and all things related in Databricks (e.g. metastore assignment, vpc network assignment) are successfully pruned, except for the underlying GKE cluster, GCS buckets that Databricks provisioned when creating the workspace.

Therefore when I would like to destroy the VPC networking resources with another target destroy, it gives out an error

Error when reading or editing Subnetwork: googleapi: Error 400: The subnetwork resource 'projects/data-platform-437607/regions/europe-west1/subnetworks/production-subnet' is already being used by 'projects/data-platform-437607/zones/europe-west1-b/instances/gke-db-4319326960483-system-pool-2024-f207051a-671q', resourceInUseByAnotherResource

This error is expected because the GCP resources are still there, but nowhere in my Terraform config had I created these resources seperately (bc they are automatically provisioned by Databricks during workspace creation).

My question is: is it possible to destroy these remaining Databricks provisioned GCP resources in Terraform or manual deletion is the only way to go? Thank you very much!

Attached is my Terraform config for workspace:

resource "databricks_mws_workspaces" "this" { provider = databricks.acc account_id = var.databricks_account_id workspace_name = var.databricks_workspace_name location = var.google_compute_subnet_region cloud_resource_container { gcp { project_id = var.google_project } } network_id = var.databricks_mws_network_id gke_config { connectivity_type = "PRIVATE_NODE_PUBLIC_MASTER" master_ip_range = var.gke_master_ip_range } token { comment = "Terraform provisioned workspace ${var.dbx_env}" } # this makes sure that the NAT is created for outbound traffic before creating the workspace depends_on = [var.google_compute_router_nat] }

#GCP #Terraform #Databricks

Re: Terraform Destroy not able to prune Databricks Provisioned GKE Cluster on GCP

HaggMan — Tue, 29 Oct 2024 16:03:50 GMT

I would guess that at this point, you will have to destroy them manually. In the future, you might want to let your TF script create all the requisite network resources; then the script will also be able to destroy them. This is a general GKE issue and not specific to Bricks. GKE does a lot of funny things at the network level and if you don't do them first, ha.

If you look at the example in the provider here: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/gcp-workspace

I think they have an example creating the network.

Re: Terraform Destroy not able to prune Databricks Provisioned GKE Cluster on GCP

KLin — Tue, 29 Oct 2024 17:04:46 GMT

Turns out it takes some time for the underlying GKE clusters to be deleted...The lesson here is that one should wait around 10 minutes between the 2 Terraform target destroys.

Thanks for the timely reply regardless 🙂

Re: Terraform Destroy not able to prune Databricks Provisioned GKE Cluster on GCP

HaggMan — Tue, 29 Oct 2024 17:25:20 GMT

Ha, that's true, too. I forget how long it takes things to delete, but I've run into it many time.

Best of luck to you!