Hi Databricks Community,
I need some guidance regarding catalogs and external locations across multiple environments. Here's my situation:
I've set up a resource group (dev-rg) and created a Databricks workspace where I successfully created catalogs (bronze, silver, gold) and external locations using storage credentials. However, I've now created a new resource group (qa-rg) with a new Databricks workspace. Surprisingly, I can see all the external locations and catalogs from the dev environment in the qa environment.
My goal is to create the same catalogs and external locations in the qa environment while keeping the same naming convention across all environments (dev, qa, and prod). However, the catalogs and external locations from the dev environment are also visible in the qa environment, and any new ones I create in qa are also visible in dev. The same issue persists in the prod environment.
I understand that I can restrict access through workspace settings to prevent cross-workspace access, but I need to ensure that catalogs and external locations are not visible across different workspaces. For example, I want to prevent the dev catalog from being visible in the qa environment and vice versa, even though there are some common users across environments.
How can I ensure that catalogs and external locations are isolated to their respective workspaces while maintaining the same naming convention across environments? Any advice or best practices for achieving this would be greatly appreciated.
Thanks in advance for your help!