Restricting Catalog and External Location Visibility Across Databricks Workspaces

[yvishal519]

Hi Databricks Community,

I need some guidance regarding catalogs and external locations across multiple environments. Here's my situation:

I've set up a resource group (dev-rg) and created a Databricks workspace where I successfully created catalogs (bronze, silver, gold) and external locations using storage credentials. However, I've now created a new resource group (qa-rg) with a new Databricks workspace. Surprisingly, I can see all the external locations and catalogs from the dev environment in the qa environment.

My goal is to create the same catalogs and external locations in the qa environment while keeping the same naming convention across all environments (dev, qa, and prod). However, the catalogs and external locations from the dev environment are also visible in the qa environment, and any new ones I create in qa are also visible in dev. The same issue persists in the prod environment.

I understand that I can restrict access through workspace settings to prevent cross-workspace access, but I need to ensure that catalogs and external locations are not visible across different workspaces. For example, I want to prevent the dev catalog from being visible in the qa environment and vice versa, even though there are some common users across environments.

How can I ensure that catalogs and external locations are isolated to their respective workspaces while maintaining the same naming convention across environments? Any advice or best practices for achieving this would be greatly appreciated.

Thanks in advance for your help!

[menotron · Accepted solution]

Hi @yvishal519, have you assigned the same metastore to both the workspace?
Ideally, that should be okay as long you have control on who has access to what workspace and use what catalog.
You can't have the same names though. Easier approach would be to name the catalogs with the environment identifier (dev_bronze, qa_silver, etc) and manage permissions in UC.

But as I understand you are trying to stand the two environments in separate resource groups and maintaining the same naming conventions, you can consider creating a separate metastore, the top-level container for metadata in Unity Catalog for each environment. 

[Kaniz_Fatma]

Hi @yvishal519, Thanks for reaching out! Please review the response and let us know if it answers your question. Your feedback is valuable to us and the community.

If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

We appreciate your participation and are here if you need further assistance!