cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Obtain access of Azure metastore storage account to configure Lifecycle management

heathwinning
New Contributor II

I recently set up an Azure Databricks workspace with an automatically created metastore and metastore-level root storage within the metastore blob storage account. All the catalogs, schemas, and tables/volumes have been created without a specified or external location, so the data all reside in the metastore blob storage account under the container named "unity-catalog-storage".

Because of the "System deny assignment created by Azure Databricks" I have no direct access to the metastore blob storage account, and therefore cannot set the access tier of some large raw files to Cool, nor can I create lifecycle management policies to do this automatically.

I regret not setting up a separate storage account for catalogs, but if possible I'd love to avoid risking migration of lots of data in lots of tables. Is there a way to achieve the access required to configure Lifecycle management?

1 REPLY 1

Thanks for your response, @Retired_mod. I already have Unity Catalog configured using an access connector and managed identity, these were automatically created by the Databricks workspace initialisation. The issue I'm facing is that [Azure Blob Lifecycle Management Policies](https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-policy-configure?tabs=azu...) require access to the storage container, but I am prevented from doing this by the Deny assignment.

I have tried

  • logging in as the managed identity that has access to the storage container, but the managed identity is also blocked by a Deny assignment.
  • creating another managed identity with access to the storage container, but the resource group is blocked by a Deny assignment

I want to know if there is a way around these Deny assignments as an administrator.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group