Obtain access of Azure metastore storage account to configure Lifecycle management

heathwinning
New Contributor II

‎08-01-2024 05:13 PM

I recently set up an Azure Databricks workspace with an automatically created metastore and metastore-level root storage within the metastore blob storage account. All the catalogs, schemas, and tables/volumes have been created without a specified or external location, so the data all reside in the metastore blob storage account under the container named "unity-catalog-storage".

Because of the "System deny assignment created by Azure Databricks" I have no direct access to the metastore blob storage account, and therefore cannot set the access tier of some large raw files to Cool, nor can I create lifecycle management policies to do this automatically.

I regret not setting up a separate storage account for catalogs, but if possible I'd love to avoid risking migration of lots of data in lots of tables. Is there a way to achieve the access required to configure Lifecycle management?

0 Kudos

heathwinning
New Contributor II
In response to Retired_mod

‎08-07-2024 02:38 PM

Thanks for your response, @Retired_mod. I already have Unity Catalog configured using an access connector and managed identity, these were automatically created by the Databricks workspace initialisation. The issue I'm facing is that [Azure Blob Lifecycle Management Policies](https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-policy-configure?tabs=azu...) require access to the storage container, but I am prevented from doing this by the Deny assignment.

I have tried

  • logging in as the managed identity that has access to the storage container, but the managed identity is also blocked by a Deny assignment.
  • creating another managed identity with access to the storage container, but the resource group is blocked by a Deny assignment

I want to know if there is a way around these Deny assignments as an administrator.

0 Kudos