10-29-2024 08:52 AM
Hi there,
newbie here in Databricks on GCP. I provisioned my Databricks workspace with Terraform and all worked well. Now when I would like to target destroy my workspace, issues occur:
When I do terraform destroy -target module.workspace, the workspace and all things related in Databricks (e.g. metastore assignment, vpc network assignment) are successfully pruned, except for the underlying GKE cluster, GCS buckets that Databricks provisioned when creating the workspace.
Therefore when I would like to destroy the VPC networking resources with another target destroy, it gives out an error
Error when reading or editing Subnetwork: googleapi: Error 400: The subnetwork resource 'projects/data-platform-437607/regions/europe-west1/subnetworks/production-subnet' is already being used by 'projects/data-platform-437607/zones/europe-west1-b/instances/gke-db-4319326960483-system-pool-2024-f207051a-671q', resourceInUseByAnotherResource
This error is expected because the GCP resources are still there, but nowhere in my Terraform config had I created these resources seperately (bc they are automatically provisioned by Databricks during workspace creation).
My question is: is it possible to destroy these remaining Databricks provisioned GCP resources in Terraform or manual deletion is the only way to go? Thank you very much!
Attached is my Terraform config for workspace:
resource "databricks_mws_workspaces" "this" {
provider = databricks.acc
account_id = var.databricks_account_id
workspace_name = var.databricks_workspace_name
location = var.google_compute_subnet_region
cloud_resource_container {
gcp {
project_id = var.google_project
}
}
network_id = var.databricks_mws_network_id
gke_config {
connectivity_type = "PRIVATE_NODE_PUBLIC_MASTER"
master_ip_range = var.gke_master_ip_range
}
token {
comment = "Terraform provisioned workspace ${var.dbx_env}"
}
# this makes sure that the NAT is created for outbound traffic before creating the workspace
depends_on = [var.google_compute_router_nat]
}
#GCP #Terraform #Databricks
10-29-2024 10:04 AM
Turns out it takes some time for the underlying GKE clusters to be deleted...The lesson here is that one should wait around 10 minutes between the 2 Terraform target destroys.
Thanks for the timely reply regardless 🙂
10-29-2024 09:03 AM
I would guess that at this point, you will have to destroy them manually. In the future, you might want to let your TF script create all the requisite network resources; then the script will also be able to destroy them. This is a general GKE issue and not specific to Bricks. GKE does a lot of funny things at the network level and if you don't do them first, ha.
If you look at the example in the provider here: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/gcp-workspace
I think they have an example creating the network.
10-29-2024 10:04 AM
Turns out it takes some time for the underlying GKE clusters to be deleted...The lesson here is that one should wait around 10 minutes between the 2 Terraform target destroys.
Thanks for the timely reply regardless 🙂
10-29-2024 10:25 AM
Ha, that's true, too. I forget how long it takes things to delete, but I've run into it many time.
Best of luck to you!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group