topic Re: Understanding the Use of a Specific Terraform Block in Unity Catalog Automation in Data Governance

Understanding the Use of a Specific Terraform Block in Unity Catalog Automation

jv_v — Tue, 25 Jun 2024 07:42:49 GMT

I am currently working on automating Unity Catalog (UC) using Terraform, and I came across the following Terraform block:

resource "databricks_metastore_data_access" "first" {

provider = databricks.Workspace

metastore_id = databricks_metastore.this.id

name = "the-metastore-key"

azure_managed_identity {

access_connector_id = azurerm_databricks_access_connector.unity.id

}

is_default = true

depends_on = [databricks_metastore_assignment.this]

}

I have a few questions regarding the use of this block:

Purpose and Functionality: Could you explain the purpose and functionality of this specific Terraform block in the context of Unity Catalog automation? How does it contribute to the overall automation process?
Mandatory or Optional: Is it mandatory to include this Terraform block in my UC automation scripts? What could be the consequences or limitations if I choose not to use it?
Provider Usage: For this block, can I use both the Databricks account-level provider and the Databricks workspace-level provider? Are there any specific scenarios or best practices where one is preferred over the other?Any insights or experiences shared would be greatly appreciated!

Re: Understanding the Use of a Specific Terraform Block in Unity Catalog Automation

giuseppegrieco — Tue, 25 Jun 2024 11:09:45 GMT

Hello,

The terraform block you've shared defines authentication methods for accessing cloud storage used as the default location for the metastore. While optional, not defining it means you won't be able to utilize the default storage location for your metastore (which serves as the default location for catalogs, schemas, and tables unless a storage location is specified at any level below the metastore one).

I hope this addresses your initial two questions. Regarding the third, a brief answer is yes, you can use either the account-level or workspace-level provider. In my preference, I lean towards the account-level provider since it isn't specifically tied to workspace resources.

For further documentation I suggest to visit https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_data_access.

Re: Understanding the Use of a Specific Terraform Block in Unity Catalog Automation

jv_v — Thu, 27 Jun 2024 14:34:30 GMT

I implemented the following Terraform code for configuring a Databricks metastore data access:

terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
}
databricks = {
source = "databricks/databricks"

}
}
}

provider "azurerm"{
alias = "azure"
skip_provider_registration = true
features {}
subscription_id = var.subscription_id
tenant_id = var.tenant_id
client_id = var.client_id
client_secret = var.client_secret
}

// Provider for databricks account
provider "databricks" {
alias = "azure_account"
host = "https://accounts.azuredatabricks.net"
account_id = var.account_id
#auth_type = "azure-cli"
client_id = var.client_id
client_secret = var.db_client_secret

}

// Provider for databricks workspace
provider "databricks" {
alias = "Workspace"
host = local.databricks_workspace_host
client_id = var.client_id
client_secret = var.db_client_secret
}

// Task: Create the first unity catalog metastore
resource "databricks_metastore" "this" {
provider = databricks.azure_account
name = var.metastore_name
region = var.use_existing_resource_group ? data.azurerm_resource_group.existing[0].location : azurerm_resource_group.new[0].location
storage_root = format("abfss://%s@%s.dfs.core.windows.net/",
azurerm_storage_container.unity_catalog.name,
azurerm_storage_account.unity_catalog.name)
force_destroy = true
owner = var.owner
}
// Task : Attach the databricks workspace to the metastore
resource "databricks_metastore_assignment" "this" {
provider = databricks.Workspace
workspace_id = local.databricks_workspace_id
metastore_id = databricks_metastore.this.id
default_catalog_name = var.default_catalog_name
}

//Task :Assign managed identity to metastore
resource "databricks_metastore_data_access" "first" {
provider = databricks.azure_account
metastore_id = databricks_metastore.this.id
name = "the-metastore-key"
azure_managed_identity {
access_connector_id = azurerm_databricks_access_connector.unity.id
}
is_default = true
depends_on = [databricks_metastore_assignment.this]
}

output "metastore_data_access_details" {
value = {
metastore_id = databricks_metastore_data_access.first.id
access_connector_id = databricks_metastore_data_access.first.azure_managed_identity
}
}

However, I'm encountering the following error when executing this code:

"databricks_metastore_data_access.first" error: cannot create metastore data access: User does not have CREATE EXTERNAL LOCATION on Metastore"

Any insights or suggestions to resolve this issue would be greatly appreciated!