cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Understanding the Use of a Specific Terraform Block in Unity Catalog Automation

jv_v
New Contributor III

I am currently working on automating Unity Catalog (UC) using Terraform, and I came across the following Terraform block:

 

 
resource "databricks_metastore_data_access" "first" {
  provider = databricks.Workspace
  metastore_id = databricks_metastore.this.id
  name         = "the-metastore-key"
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.unity.id
  }
  is_default = true
  depends_on = [databricks_metastore_assignment.this]
}

I have a few questions regarding the use of this block:

  1. Purpose and Functionality: Could you explain the purpose and functionality of this specific Terraform block in the context of Unity Catalog automation? How does it contribute to the overall automation process?

  2. Mandatory or Optional: Is it mandatory to include this Terraform block in my UC automation scripts? What could be the consequences or limitations if I choose not to use it?

  3. Provider Usage: For this block, can I use both the Databricks account-level provider and the Databricks workspace-level provider? Are there any specific scenarios or best practices where one is preferred over the other?Any insights or experiences shared would be greatly appreciated!

1 ACCEPTED SOLUTION

Accepted Solutions

giuseppegrieco
New Contributor III

Hello,

The terraform block you've shared defines authentication methods for accessing cloud storage used as the default location for the metastore. While optional, not defining it means you won't be able to utilize the default storage location for your metastore (which serves as the default location for catalogs, schemas, and tables unless a storage location is specified at any level below the metastore one).

I hope this addresses your initial two questions. Regarding the third, a brief answer is yes, you can use either the account-level or workspace-level provider. In my preference, I lean towards the account-level provider since it isn't specifically tied to workspace resources.

For further documentation I suggest to visit https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_data_a....

View solution in original post

2 REPLIES 2

giuseppegrieco
New Contributor III

Hello,

The terraform block you've shared defines authentication methods for accessing cloud storage used as the default location for the metastore. While optional, not defining it means you won't be able to utilize the default storage location for your metastore (which serves as the default location for catalogs, schemas, and tables unless a storage location is specified at any level below the metastore one).

I hope this addresses your initial two questions. Regarding the third, a brief answer is yes, you can use either the account-level or workspace-level provider. In my preference, I lean towards the account-level provider since it isn't specifically tied to workspace resources.

For further documentation I suggest to visit https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_data_a....

jv_v
New Contributor III

I implemented the following Terraform code for configuring a Databricks metastore data access:

terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
}
databricks = {
source = "databricks/databricks"

}
}
}

provider "azurerm"{
alias = "azure"
skip_provider_registration = true
features {}
subscription_id = var.subscription_id
tenant_id = var.tenant_id
client_id = var.client_id
client_secret = var.client_secret
}

// Provider for databricks account
provider "databricks" {
alias = "azure_account"
host = "https://accounts.azuredatabricks.net"
account_id = var.account_id
#auth_type = "azure-cli"
client_id = var.client_id
client_secret = var.db_client_secret

}

// Provider for databricks workspace
provider "databricks" {
alias = "Workspace"
host = local.databricks_workspace_host
client_id = var.client_id
client_secret = var.db_client_secret
}

 

// Task: Create the first unity catalog metastore
resource "databricks_metastore" "this" {
provider = databricks.azure_account
name = var.metastore_name
region = var.use_existing_resource_group ? data.azurerm_resource_group.existing[0].location : azurerm_resource_group.new[0].location
storage_root = format("abfss://%s@%s.dfs.core.windows.net/",
azurerm_storage_container.unity_catalog.name,
azurerm_storage_account.unity_catalog.name)
force_destroy = true
owner = var.owner
}
// Task : Attach the databricks workspace to the metastore
resource "databricks_metastore_assignment" "this" {
provider = databricks.Workspace
workspace_id = local.databricks_workspace_id
metastore_id = databricks_metastore.this.id
default_catalog_name = var.default_catalog_name
}

//Task :Assign managed identity to metastore
resource "databricks_metastore_data_access" "first" {
provider = databricks.azure_account
metastore_id = databricks_metastore.this.id
name = "the-metastore-key"
azure_managed_identity {
access_connector_id = azurerm_databricks_access_connector.unity.id
}
is_default = true
depends_on = [databricks_metastore_assignment.this]
}

output "metastore_data_access_details" {
value = {
metastore_id = databricks_metastore_data_access.first.id
access_connector_id = databricks_metastore_data_access.first.azure_managed_identity
}
}

However, I'm encountering the following error when executing this code:

"databricks_metastore_data_access.first" error: cannot create metastore data access: User does not have CREATE EXTERNAL LOCATION on Metastore"

Any insights or suggestions to resolve this issue would be greatly appreciated!
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!