cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

External locations being shared across workspaces

Debi-Moha
New Contributor II

Currently, we have 3 Unity Catalog enabled workspaces sharing the same metastore. Now, when we create an external location or storage credential in any of the workspaces, it gets reflected across all workspaces. We are looking for some best practices around setting up external locations, such that in case we have a replicating setup like this, it would not have any security concerns. I can elaborate on this further if required, and I would appreciate any inputs on this setup or our approach around this.

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @Debi-MohaSetting up external locations in a replicating setup like yours requires careful planning to ensure data governance and security.

Here are some best practices for managing external locations in Unity Catalog:

  1. Understand Data Governance Building Blocks:

    • Unity Catalog operates within a hierarchy of objects: Metastore, Catalog, and Schema.
    • Metastore: Top-level container for objects, managing data assets (tables, views, volumes) and permissions. It lives at the account level and provides regional isolation.
    • Catalog: Represents a logical grouping of schemas. It’s the primary unit of data isolation. You can have catalogs for different data access requirements (e.g., production vs. development data).
    • Schema: Contains tables, views, and volumes within a catalog.
  2. Create Separate Catalogs:

    • Use catalogs to isolate data. Each catalog can mirror organizational units or software development lifecycle scopes.
    • For example, create separate catalogs for production data, development data, or sensitive customer data.
    • Avoid overlapping data paths by organizing external tables and volumes within sub-directories of catalogs.
  3. Configure External Locations Properly:

    • Migrate mounts on cloud storage locations to external locations in Unity Catalog using Catalog Explorer.
    • Never create an external volume or table at the root of an external location. Instead, create them within sub-directories.
    • Situate external locations at the base of storage containers to avoid path overlaps.
  4. Storage Credentials:

    • Assign appropriate storage credentials to external locations.
    • Ensure that only authorized users have access to these credentials.
    • Regularly review and update credentials to maintain security.
  5. Audit and Monitor:

    • Unity Catalog provides centralized administration and auditing of data access.
    • Regularly review access logs and permissions to detect any anomalies.
    • Monitor external locations for unauthorized changes.
  6. Data Isolation and Replication:

    • While metastores provide regional isolation, data isolation should begin at the catalog level.
    • Consider the impact of replication: External locations and storage credentials will be shared across workspaces.
    • Ensure that replication doesn’t compromise security or data integrity.

If you need further assistance or have additional details to share, feel free to elaborate, and I’ll be happy to provide more insights! 🌟

AlliaKhosla
New Contributor III
New Contributor III

Hi @Debi-Moha  Currently we do not have a mechanism to isolate the external locations and storage credentials based on workspaces, since the metastore is shared across the workspaces.

Please check below document for recommendations on securing external locations:

https://docs.databricks.com/en/data-governance/unity-catalog/best-practices.html#recommendations-for...

We do have a Feature request for this and this feature is considered for future.

Ideas Portal Idea:DB-I-7138

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!