I'm currently working with a banking customer migrating from Hive Metastore to Unity Catalog. While planning their catalog layout, one of their platform engineers asked the question that prompted this post: for their schemas and volumes, should they use managed or external storage, and how would they keep each business unit's data in its own storage account?

They were leaning toward external, for two reasons: they believed managed storage meant Databricks would take custody of their data, and they believed managed storage could not be physically segregated by team. I hear both on almost every Hive Metastore migration. Neither holds up. The data stays in your own Azure storage either way, and managed storage can be segregated per business unit right down to the schema. Walking through the differences gave them a clear picture of the trade-offs, and since the same question comes up so often, I figured sharing it more widely would help.

This post is that walkthrough. It covers both storage models on Azure: what they are, their trade-offs, when to pick each, where the two misconceptions break down, and a tested end-to-end example you can run yourself. Everything is backed by the Azure Databricks documentation linked at the end.

1. The three building blocks

Before comparing the two models, it helps to separate three concepts that are often conflated.

2. Managed vs external: the core difference

The difference comes down to who controls the physical storage layout and lifecycle. Both store data in your own Azure storage account. The table below compares them side by side.

3. Myth 1: "Managed means lock-in"

The word "managed" is easy to misread as "Databricks takes custody of your data" or "you can never get it back out." Neither is true.

• Your data stays in your own Azure storage. Managed tables and volumes are written to an ADLS Gen2 account in your Azure subscription, governed by your network and encryption policies. Databricks does not copy your data into Databricks-owned storage. You designate the storage account; UC writes there.
• "Managed" refers to file management, not ownership. It means UC handles the physical file layout, compaction, statistics, and cleanup for you. You still own the storage account, the data, and the keys.
• The format is open, so there is no lock-in. Data is stored as open Delta Lake (Parquet) files, readable from outside Databricks through open interfaces: the Iceberg REST Catalog, Delta Sharing, and credential vending for engines such as Spark, Trino, DuckDB, and Snowflake. You can share it without copying.
• You keep full access control. Every read and write goes through Unity Catalog privileges that you define.

A more accurate mental label for "managed" is "Databricks-optimized storage in your own account," as opposed to "Databricks-owned storage."

4. Myth 2: "Managed storage can't be segregated"

This is the one that pushes teams toward external storage unnecessarily. A managed table or volume does not all land in one big shared bucket by default. UC resolves the managed storage path in this order:

1. Schema managed location, if set.
2. Otherwise the catalog managed location, if set.
3. Otherwise the metastore managed location.

This gives you three levels of granularity to physically separate managed data. You can point different catalogs, or even different schemas, at completely different ADLS Gen2 containers or paths.

Metastore managed location:  abfss://metastore@examplestorage.dfs.core.windows.net/
  └── Catalog: sales          MANAGED LOCATION  abfss://sales@examplestorage.dfs.core.windows.net/
        └── Schema: orders      MANAGED LOCATION  abfss://sales-orders@examplestorage.dfs.core.windows.net/

So if the requirement is separate underlying storage per team or per sensitivity tier, managed storage meets it. You get clean physical isolation by assigning each catalog or schema its own managed location. What managed storage does not give you is control over the per-table directory names: UC creates hashed subdirectories (under __unitystorage/...) to guarantee uniqueness. You control the container/path boundary, UC controls the layout inside it.

Volumes work the same way: create separate managed volumes in separate schemas or catalogs, each with its own managed location, to segregate file storage exactly as you would with separate external paths.

5. Managed storage: pros and cons

Pros

• Automatic optimization. Predictive optimization runs OPTIMIZE, VACUUM, and ANALYZE for you. On by default for accounts created after November 11, 2024, rolling out to older accounts.
• Automatic liquid clustering. UC can select and maintain clustering keys automatically.
• Lower long-term cost and faster queries. Intelligent file sizing, statistics, and metadata caching reduce both storage and compute.
• Less operational burden. No path management, no manual layout decisions, no orphaned-file cleanup.
• Auto-upgrade to new features. Managed tables pick up new platform capabilities (catalog commits, multi-statement transactions, full-text search indexes) without re-architecting.
• Strongest governance posture. No path-based access means UC privileges are always enforced. There is no side door to the files.

Cons

• No direct path access. You cannot point a non-Databricks tool at the files by path. Access is through UC-governed interfaces only (which now include the Iceberg REST Catalog, Delta Sharing, and credential vending for engines like Spark, Trino, DuckDB, and Snowflake).
• You do not control the directory layout. UC owns the subdirectory structure.
• Drop deletes data (after the undrop window). This is a plus for hygiene, a minus if you rely on files persisting after a drop.

6. External storage: pros and cons

Pros

• Explicit path control. Data lives exactly where you put it. Useful when a downstream system expects a fixed path.
• Interoperability with non-Databricks tools. External tools can read and write the same files directly at the path.
• Register existing data in place. Point UC at data that already exists without moving it.
• Drop does not delete data. The files survive a dropped table or volume.

Cons

• No automatic optimization. Predictive optimization, auto liquid clustering, and the managed-table maintenance features do not apply. You own OPTIMIZE, VACUUM, and statistics.
• You own the lifecycle. Orphaned files, layout, and cleanup are your responsibility.
• Governance gap on direct access. When external tools read files by path outside UC interfaces, UC privileges are not enforced on that direct file-system access. Governance is only as strong as the storage-account controls.
• More moving parts. Every external path needs a storage credential, an external location, and grants. More to provision and audit.
• Metadata drift. Changes made to files outside Databricks may require MSCK REPAIR TABLE ... SYNC METADATA.

7. When to choose what

A typical multi-team environment ends up with managed as the default for governed analytical tables, and external locations reserved for ingestion landing zones, file interchange, and integration with tools that genuinely need direct path access.

8. Recommended best practices

1. Default to managed. Make managed tables and volumes the norm. Reach for external only when a concrete requirement (path access, external tooling, register-in-place) demands it.
2. Segregate with catalogs and schemas, not with external storage. Assign each team or environment its own catalog with its own MANAGED LOCATION. Use schema-level managed locations for finer isolation. This gives you physical separation while keeping the governance benefits of managed.
3. Keep managed and external storage in separate, dedicated containers. Managed storage cannot overlap external tables or external volumes. Plan a clean container layout up front: one set of containers for managed, separate containers for external/landing data.
4. One Access Connector and storage credential per trust boundary. Do not share a single credential across unrelated teams if their storage accounts have different access policies. Scope credentials to match your isolation model.
5. Prefer volumes over raw external-location file access. When teams need file access, govern it through a volume rather than granting READ FILES / WRITE FILES broadly on an external location.
6. Grant the Access Connector identity Storage Blob Data Contributor on the target ADLS Gen2 account or container. Nothing works without this role.
7. Use workspace bindings on external locations and credentials to restrict which workspaces can use them, if you run multiple workspaces.

9. Creating external locations on Azure

If you decide you need external storage, here is the end-to-end setup.

Permissions required

Creating external storage is a two-step, two-privilege process.

A non-admin who needs to create external locations must be granted, by a metastore admin: CREATE STORAGE CREDENTIAL on the metastore (if they will also create the credential), CREATE EXTERNAL LOCATION on the metastore, and CREATE EXTERNAL LOCATION (or MANAGE) on the specific storage credential.

Azure prerequisites

1. An ADLS Gen2 storage account with hierarchical namespace enabled. WORM/immutable containers are not supported.
2. A Databricks Access Connector for Azure with a system-assigned (or user-assigned) managed identity.
3. That managed identity granted the Storage Blob Data Contributor role on the storage account or container.

Step 1: Create the storage credential

On Azure, storage credentials backed by a managed identity are created through Catalog Explorer, the Databricks CLI, the REST API, or Terraform (there is no inline SQL form for Azure managed-identity credentials). The CLI form is:

databricks storage-credentials create --json '{
  "name": "adls_cred",
  "comment": "Managed identity for ADLS Gen2",
  "azure_managed_identity": {
    "access_connector_id": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Databricks/accessConnectors/<connector-name>"
  }
}'

If the Access Connector uses a user-assigned managed identity, add inside azure_managed_identity:

"managed_identity_id": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<mi-name>"

In Catalog Explorer the equivalent fields are: Credential Type = "Azure Managed Identity", "Access Connector ID", and optionally "Managed Identity ID".

Step 2: Create the external location

CREATE EXTERNAL LOCATION IF NOT EXISTS sales_landing
  URL 'abfss://landing@examplestorage.dfs.core.windows.net/sales'
  WITH (STORAGE CREDENTIAL adls_cred)
  COMMENT 'Sales landing zone';

Notes: the path is abfss://<container>@<storage-account>.dfs.core.windows.net/<path>. Names containing hyphens must be backtick-quoted. The keyword STORAGE is optional but the canonical form is WITH (STORAGE CREDENTIAL ...).

Step 3: Grant access on the external location

-- Allow a team to create external tables on this location
GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION sales_landing TO `data-engineers`;

-- Allow creating external volumes
GRANT CREATE EXTERNAL VOLUME ON EXTERNAL LOCATION sales_landing TO `data-engineers`;

-- Direct file read/write (WRITE FILES requires READ FILES too)
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION sales_landing TO `ingestion-svc`;

-- Allow this location (or a subpath) to back a managed location for a catalog/schema
GRANT CREATE MANAGED STORAGE ON EXTERNAL LOCATION sales_landing TO `platform-admins`;

-- Review and transfer ownership
SHOW GRANTS ON EXTERNAL LOCATION sales_landing;
ALTER EXTERNAL LOCATION sales_landing OWNER TO `platform-admins`;

Defining a segregated managed location on top of external

Because catalog and schema managed locations must sit inside an external location, the pattern for segregated managed storage is: create the external location once, then point each catalog or schema's managed location at a subpath of it.

-- Catalog-level segregation: Sales managed data in its own container
CREATE CATALOG IF NOT EXISTS sales
  MANAGED LOCATION 'abfss://sales@examplestorage.dfs.core.windows.net/managed';

-- Schema-level segregation: Orders data in a separate path
CREATE SCHEMA IF NOT EXISTS sales.orders
  MANAGED LOCATION 'abfss://sales-orders@examplestorage.dfs.core.windows.net/managed';

-- A managed volume inherits the schema's managed location, physically isolated
CREATE VOLUME IF NOT EXISTS sales.orders.landing_files;

10. Hands-on walkthrough: provision it and watch the data land

Reading about resolution order is one thing. Watching a managed table land in the exact container you chose is more convincing. Here is the full sequence run end to end on an Azure workspace, with the actual output. It creates a catalog on one container and a schema on a different container, then shows that the managed table and volume follow the schema.

Azure side: storage account and access connector

RG=rg-uc-blog; LOC=eastus; SA=ucblogdemostore

az group create -n $RG -l $LOC

# ADLS Gen2 = StorageV2 with hierarchical namespace enabled
az storage account create -n $SA -g $RG -l $LOC \
  --sku Standard_LRS --kind StorageV2 --enable-hierarchical-namespace true

for c in sales sales-orders landing; do
  az storage container create --account-name $SA --name $c --auth-mode login
done

# Databricks Access Connector with a system-assigned managed identity
az databricks access-connector create -n ac-uc-blog -g $RG -l $LOC --identity-type SystemAssigned

# grant that managed identity access to the storage account
MI=$(az databricks access-connector show -n ac-uc-blog -g $RG --query identity.principalId -o tsv)
SAID=$(az storage account show -n $SA -g $RG --query id -o tsv)
az role assignment create --assignee-object-id $MI --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" --scope "$SAID"

Unity Catalog side: credential and external locations

# storage credential wrapping the access connector
databricks storage-credentials create --json '{
  "name": "adls_cred",
  "azure_managed_identity": { "access_connector_id": "<access-connector-resource-id>" }
}'

# one external location per container
databricks external-locations create sales_el        abfss://sales@ucblogdemostore.dfs.core.windows.net/        adls_cred
databricks external-locations create sales_orders_el abfss://sales-orders@ucblogdemostore.dfs.core.windows.net/ adls_cred
databricks external-locations create landing_el      abfss://landing@ucblogdemostore.dfs.core.windows.net/      adls_cred

Create the catalog and a schema on a different container

This is the segregation step: the catalog's managed location is on the sales container, but the schema declares its own managed location on sales-orders.

CREATE CATALOG av_sales_demo
  MANAGED LOCATION 'abfss://sales@ucblogdemostore.dfs.core.windows.net/managed';

CREATE SCHEMA av_sales_demo.orders
  MANAGED LOCATION 'abfss://sales-orders@ucblogdemostore.dfs.core.windows.net/managed';

CREATE TABLE av_sales_demo.orders.line_items (order_id BIGINT, region STRING, amount DECIMAL(10,2));
INSERT INTO av_sales_demo.orders.line_items VALUES (1,'EMEA',120.50),(2,'AMER',88.00),(3,'APAC',310.75);

CREATE VOLUME av_sales_demo.orders.landing_files;                                    -- managed
CREATE EXTERNAL VOLUME av_sales_demo.orders.raw_drop
  LOCATION 'abfss://landing@ucblogdemostore.dfs.core.windows.net/sales/raw_drop';    -- external

Prove where the data physically landed

DESCRIBE DETAIL  av_sales_demo.orders.line_items;
DESCRIBE VOLUME  av_sales_demo.orders.landing_files;
DESCRIBE VOLUME  av_sales_demo.orders.raw_drop;

Actual output (trimmed to the relevant fields):

-- line_items  (MANAGED table)
location: abfss://sales-orders@ucblogdemostore.../managed/__unitystorage/schemas/a412770d-.../tables/5d24725c-...

-- landing_files  (MANAGED volume)
volume_type:      MANAGED
storage_location: abfss://sales-orders@ucblogdemostore.../managed/__unitystorage/schemas/a412770d-.../volumes/d3235d73-...

-- raw_drop  (EXTERNAL volume)
volume_type:      EXTERNAL
storage_location: abfss://landing@ucblogdemostore.../sales/raw_drop

The catalog's managed location is on the sales container, yet the managed table and managed volume both landed under sales-orders, because the orders schema declared its own managed location and the schema wins over the catalog. That is physical storage segregation achieved with fully managed objects: no external tables, no loss of automatic optimization. The external volume sits exactly at the path it was given.

The same view in Catalog Explorer

Key takeaways

1. Storage credential, external location, and managed location are three distinct things. A credential is how to authenticate, an external location is which path it governs, a managed location is where managed objects land.
2. Managed is the default and the recommendation for governed analytical tables and volumes: automatic optimization, lower cost, strongest governance, least maintenance.
3. Managed does not mean lock-in. Your data stays in your own Azure storage, in open Delta format, reachable through open interfaces.
4. Managed storage can be physically segregated by assigning per-catalog or per-schema managed locations. The resolution order is schema > catalog > metastore.
5. You control the container boundary, UC controls the layout inside it. If a requirement is about the container or path boundary, managed meets it. If it is about exact directory naming or external-tool path access, that is where external fits.
6. External is the right tool for landing zones, file interchange, register-in-place, and non-Databricks engines. It costs you the automatic optimizations and opens a path-access governance gap, so scope it deliberately.
7. To create external storage you need two privileges: CREATE STORAGE CREDENTIAL for the credential, and CREATE EXTERNAL LOCATION on both the metastore and the credential for the location.
8. Keep managed and external in separate dedicated containers. They cannot overlap, so plan the container layout before you start.

References

Official Azure Databricks documentation:

• Specify a managed storage location in Unity Catalog: metastore, catalog, and schema managed locations; resolution order.
• Managed tables: default table type, automatic optimization, lifecycle.
• External tables: when to use, path control, lifecycle responsibilities.
• Create an external location: concepts and requirements.
• Manage external locations: grants (READ FILES, WRITE FILES, CREATE EXTERNAL TABLE/VOLUME, CREATE MANAGED STORAGE).
• Create a storage credential for ADLS: UI, CLI, API, and Terraform paths.
• Use Azure managed identities with the Access Connector: access connector and Storage Blob Data Contributor role.
• CREATE EXTERNAL LOCATION (SQL reference)
• CREATE CATALOG (SQL reference)
• CREATE SCHEMA (SQL reference)
• Unity Catalog privileges and securable objects