Databricks cluster pool deployed through Terraform does not have UC enabled

[erigaud]

Hello everyone,

we have a workspace with UC enabled, we already have a couple of catalogs attached and when using our personal compute we are able to read/write tables in those catalogs.

However for our jobs we deployed a cluster pool using Terraform but those clusters don't seem to have UC enabled and thus cannot access our catalogs. Did someone ever run into this issue ? 

Thank you 

[Alberto_Umana]

Hi @erigaud,

Could you please share your cluster pool configuration?

[saurabh18cs]

try by adding this to your terraform code:

data_security_mode      = "SINGLE_USER"

[erigaud]

Hello @Alberto_Umana @saurabh18cs thank you for replying, here is the configuration we're using in the terraform code : 

resource "databricks_instance_pool" "instance_pool" {
  instance_pool_name = "instance-pool-${var.environment}-${var.project}"
  min_idle_instances = 0
  max_capacity       = 30
  node_type_id       = "Standard_DS3_v2"
  idle_instance_autotermination_minutes = 30
  disk_spec {
    disk_size  = 80
    disk_count = 1
    disk_type {
      azure_disk_volume_type = "STANDARD_LRS"
    }
  }
}

Should we add something to this configuration so it works ? 

Thanks !

[saurabh18cs · Accepted solution]

hi @erigaud 

you should add this to databricks_job which is also using your instance pool.

example::

resource "databricks_job" "this" {

  new_cluster {

    instance_pool_id        = databricks_instance_pool.executor[each.key].id

    driver_instance_pool_id = databricks_instance_pool.driver[each.key].id

    spark_version           = each.value.new_cluster.spark_version

    spark_conf              = each.value.new_cluster.spark_conf

    # custom_tags             = each.value.new_cluster.custom_tags #removed tags from job level and added to instance pool level (could not have tags at job level and instance pool level at the same time)

    runtime_engine          = can(each.value.new_cluster.runtime_engine) ? each.value.new_cluster.runtime_engine == "PHOTON" ? "PHOTON" : null : null

    spark_env_vars          = each.value.new_cluster.spark_env_vars

    data_security_mode      = "SINGLE_USER" 

    azure_attributes {

      availability = can(each.value.new_cluster.azure_attributes.availability) ? each.value.new_cluster.azure_attributes.availability : null

    }

    num_workers = can(each.value.new_cluster.num_workers) ? each.value.new_cluster.num_workers : null

    dynamic "autoscale" {

      for_each    = can(each.value.new_cluster.num_workers) ? [] : [each.value.new_cluster.autoscale]

      content {

        min_workers = autoscale.value.min_workers

        max_workers = autoscale.value.max_workers

      }

    }

  }

}

[erigaud]

Confirmed that this works ! THank you