cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Restricting Catalog and External Location Visibility Across Databricks Workspaces

yvishal519
Contributor

Hi Databricks Community,

I need some guidance regarding catalogs and external locations across multiple environments. Here's my situation:

I've set up a resource group (dev-rg) and created a Databricks workspace where I successfully created catalogs (bronze, silver, gold) and external locations using storage credentials. However, I've now created a new resource group (qa-rg) with a new Databricks workspace. Surprisingly, I can see all the external locations and catalogs from the dev environment in the qa environment.

My goal is to create the same catalogs and external locations in the qa environment while keeping the same naming convention across all environments (dev, qa, and prod). However, the catalogs and external locations from the dev environment are also visible in the qa environment, and any new ones I create in qa are also visible in dev. The same issue persists in the prod environment.

I understand that I can restrict access through workspace settings to prevent cross-workspace access, but I need to ensure that catalogs and external locations are not visible across different workspaces. For example, I want to prevent the dev catalog from being visible in the qa environment and vice versa, even though there are some common users across environments.

How can I ensure that catalogs and external locations are isolated to their respective workspaces while maintaining the same naming convention across environments? Any advice or best practices for achieving this would be greatly appreciated.

Thanks in advance for your help!

1 ACCEPTED SOLUTION

Accepted Solutions

menotron
Valued Contributor

Hi @yvishal519, have you assigned the same metastore to both the workspace?
Ideally, that should be okay as long you have control on who has access to what workspace and use what catalog.
You can't have the same names though. Easier approach would be to name the catalogs with the environment identifier (dev_bronze, qa_silver, etc) and manage permissions in UC.

But as I understand you are trying to stand the two environments in separate resource groups and maintaining the same naming conventions, you can consider creating a separate metastore, the top-level container for metadata in Unity Catalog for each environment. 

View solution in original post

1 REPLY 1

menotron
Valued Contributor

Hi @yvishal519, have you assigned the same metastore to both the workspace?
Ideally, that should be okay as long you have control on who has access to what workspace and use what catalog.
You can't have the same names though. Easier approach would be to name the catalogs with the environment identifier (dev_bronze, qa_silver, etc) and manage permissions in UC.

But as I understand you are trying to stand the two environments in separate resource groups and maintaining the same naming conventions, you can consider creating a separate metastore, the top-level container for metadata in Unity Catalog for each environment. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group