Databricks setup with Azure storage

marcusfox — Wed, 10 Jan 2024 14:24:59 GMT

Hi ,

We have an issue with our intial setup and design.

We are using a single Azure, Premium, Block blob storage account with hierarchical namespace and LRS enabled.

We have three containers within it, one for each environment – Dev -Test – Prod

But the issue we encounter is that when we try to add our secondary Databricks workspace (Test) and connect into the Unity catalog we cannot target the Test container within the storage account.

So this new test workspace can only view the Dev container (The DEV Workspace was setup first) and not its own Test container.

Is there a way to have all three of our databricks workspaces looking at a single azure storage account and targeting their respective containers within the storage account?

Test WS > Test container

DEV WS > Dev container

Prd WS > container

As you know we cannot use more than one metastore in the same region so we cant use a separate storage account for each environment as a workaround.

any help here would be great.

thanks

Re: Databricks setup with Azure storage

Debayan — Thu, 18 Jan 2024 04:39:10 GMT

Hi, could you please check https://community.databricks.com/t5/data-governance/metastore-one-per-account-region-limitation/td-p/41097 and let us know if this discussion helps?

topic Re: Databricks setup with Azure storage in Get Started Discussions

Databricks setup with Azure storage

Re: Databricks setup with Azure storage