topic Re: Databricks cluster pool deployed through Terraform does not have UC enabled in Administration & Architecture

Databricks cluster pool deployed through Terraform does not have UC enabled

erigaud — Tue, 14 Jan 2025 17:02:22 GMT

Hello everyone,

we have a workspace with UC enabled, we already have a couple of catalogs attached and when using our personal compute we are able to read/write tables in those catalogs.

However for our jobs we deployed a cluster pool using Terraform but those clusters don't seem to have UC enabled and thus cannot access our catalogs. Did someone ever run into this issue ?

Thank you

Re: Databricks cluster pool deployed through Terraform does not have UC enabled

Alberto_Umana — Tue, 14 Jan 2025 17:07:51 GMT

Hi @erigaud,

Could you please share your cluster pool configuration?

Re: Databricks cluster pool deployed through Terraform does not have UC enabled

saurabh18cs — Wed, 15 Jan 2025 15:52:44 GMT

try by adding this to your terraform code:

data_security_mode = "SINGLE_USER"

Re: Databricks cluster pool deployed through Terraform does not have UC enabled

erigaud — Thu, 16 Jan 2025 08:43:11 GMT

Hello @Alberto_Umana @saurabh18cs thank you for replying, here is the configuration we're using in the terraform code :

resource "databricks_instance_pool" "instance_pool" {
  instance_pool_name = "instance-pool-${var.environment}-${var.project}"
  min_idle_instances = 0
  max_capacity       = 30
  node_type_id       = "Standard_DS3_v2"
  idle_instance_autotermination_minutes = 30
  disk_spec {
    disk_size  = 80
    disk_count = 1
    disk_type {
      azure_disk_volume_type = "STANDARD_LRS"
    }
  }
}

Should we add something to this configuration so it works ?

Thanks !

Re: Databricks cluster pool deployed through Terraform does not have UC enabled

saurabh18cs — Thu, 16 Jan 2025 09:58:54 GMT

hi @erigaud

you should add this to databricks_job which is also using your instance pool.

example::

resource "databricks_job" "this" {

new_cluster {

instance_pool_id = databricks_instance_pool.executor[each.key].id

driver_instance_pool_id = databricks_instance_pool.driver[each.key].id

spark_version = each.value.new_cluster.spark_version

spark_conf = each.value.new_cluster.spark_conf

# custom_tags = each.value.new_cluster.custom_tags #removed tags from job level and added to instance pool level (could not have tags at job level and instance pool level at the same time)

runtime_engine = can(each.value.new_cluster.runtime_engine) ? each.value.new_cluster.runtime_engine == "PHOTON" ? "PHOTON" : null : null

spark_env_vars = each.value.new_cluster.spark_env_vars

data_security_mode = "SINGLE_USER"

azure_attributes {

availability = can(each.value.new_cluster.azure_attributes.availability) ? each.value.new_cluster.azure_attributes.availability : null

}

num_workers = can(each.value.new_cluster.num_workers) ? each.value.new_cluster.num_workers : null

dynamic "autoscale" {

for_each = can(each.value.new_cluster.num_workers) ? [] : [each.value.new_cluster.autoscale]

content {

min_workers = autoscale.value.min_workers

max_workers = autoscale.value.max_workers

}

Re: Databricks cluster pool deployed through Terraform does not have UC enabled

erigaud — Wed, 22 Jan 2025 15:59:52 GMT

Confirmed that this works ! THank you