External locations being shared across workspaces

[Debi-Moha]

Currently, we have 3 Unity Catalog enabled workspaces sharing the same metastore. Now, when we create an external location or storage credential in any of the workspaces, it gets reflected across all workspaces. We are looking for some best practices around setting up external locations, such that in case we have a replicating setup like this, it would not have any security concerns. I can elaborate on this further if required, and I would appreciate any inputs on this setup or our approach around this.

[Kaniz]

Hi @Debi-Moha, Setting up external locations in a replicating setup like yours requires careful planning to ensure data governance and security.

Here are some best practices for managing external locations in Unity Catalog:

1. Understand Data Governance Building Blocks:

   • Unity Catalog operates within a hierarchy of objects: Metastore, Catalog, and Schema.
   • Metastore: Top-level container for objects, managing data assets (tables, views, volumes) and permissions. It lives at the account level and provides regional isolation.
   • Catalog: Represents a logical grouping of schemas. It’s the primary unit of data isolation. You can have catalogs for different data access requirements (e.g., production vs. development data).
   • Schema: Contains tables, views, and volumes within a catalog.

2. Create Separate Catalogs:

   • Use catalogs to isolate data. Each catalog can mirror organizational units or software development lifecycle scopes.
   • For example, create separate catalogs for production data, development data, or sensitive customer data.
   • Avoid overlapping data paths by organizing external tables and volumes within sub-directories of catalogs.

3. Configure External Locations Properly:

   • Migrate mounts on cloud storage locations to external locations in Unity Catalog using Catalog Explorer.
   • Never create an external volume or table at the root of an external location. Instead, create them within sub-directories.
   • Situate external locations at the base of storage containers to avoid path overlaps.

4. Storage Credentials:

   • Assign appropriate storage credentials to external locations.
   • Ensure that only authorized users have access to these credentials.
   • Regularly review and update credentials to maintain security.

5. Audit and Monitor:

   • Unity Catalog provides centralized administration and auditing of data access.
   • Regularly review access logs and permissions to detect any anomalies.
   • Monitor external locations for unauthorized changes.

6. Data Isolation and Replication:

   • While metastores provide regional isolation, data isolation should begin at the catalog level.
   • Consider the impact of replication: External locations and storage credentials will be shared across workspaces.
   • Ensure that replication doesn’t compromise security or data integrity.

If you need further assistance or have additional details to share, feel free to elaborate, and I’ll be happy to provide more insights! 🌟

[AlliaKhosla]

Hi @Debi-Moha  Currently we do not have a mechanism to isolate the external locations and storage credentials based on workspaces, since the metastore is shared across the workspaces.

Please check below document for recommendations on securing external locations:

https://docs.databricks.com/en/data-governance/unity-catalog/best-practices.html#recommendations-for...

We do have a Feature request for this and this feature is considered for future.

Ideas Portal Idea:DB-I-7138