Managed vs External Storage in Unity Catalog on Azure: Where Should Your Data Live?

Ashwin_DSA — Mon, 29 Jun 2026 16:46:42 GMT

I'm currently working with a banking customer migrating from Hive Metastore to Unity Catalog. While planning their catalog layout, one of their platform engineers asked the question that prompted this post: for their schemas and volumes, should they use managed or external storage, and how would they keep each business unit's data in its own storage account?

They were leaning toward external, for two reasons: they believed managed storage meant Databricks would take custody of their data, and they believed managed storage could not be physically segregated by team. I hear both on almost every Hive Metastore migration. Neither holds up. The data stays in your own Azure storage either way, and managed storage can be segregated per business unit right down to the schema. Walking through the differences gave them a clear picture of the trade-offs, and since the same question comes up so often, I figured sharing it more widely would help.

This post is that walkthrough. It covers both storage models on Azure: what they are, their trade-offs, when to pick each, where the two misconceptions break down, and a tested end-to-end example you can run yourself. Everything is backed by the Azure Databricks documentation linked at the end.

Key Takeaways

"Managed" means Databricks optimizes your files. It does not mean Databricks owns your data. Your data stays in your own Azure storage, in open Delta format.
Managed storage can be physically segregated per team or environment by assigning per-catalog or per-schema managed locations. The resolution order is schema > catalog > metastore.
Managed is the default recommendation for governed analytical tables and volumes. External is the right tool for landing zones, fixed paths, register-in-place, and non-Databricks engines.
Creating external storage requires two separate privileges: CREATE STORAGE CREDENTIAL on the metastore, and CREATE EXTERNAL LOCATION on both the metastore and the credential.

What's in this post

The three building blocks
Managed vs external: the core difference
Myth 1: "Managed means lock-in"
Myth 2: "Managed storage can't be segregated"
Managed storage: pros and cons
External storage: pros and cons
When to choose what
Recommended best practices
Creating external locations on Azure
Hands-on walkthrough: provision it and watch the data land
References

1. The three building blocks

Before comparing the two models, it helps to separate three concepts that are often conflated.

Concept	What it is	Scope
Storage credential	A securable that holds the authentication to your Azure storage. On Azure this is an Azure Managed Identity attached to a Databricks Access Connector.	Metastore
External location	A securable that pairs a storage credential with a specific `abfss://` path. It is the governed gateway to a path in ADLS Gen2.	Metastore
Managed location	A path (which must sit inside an external location) where UC stores managed tables and volumes. Can be set at metastore, catalog, or schema level.	Metastore / Catalog / Schema

The mental model: a storage credential says how to authenticate, an external location says which path that credential governs, and a managed location designates a path where UC takes over file layout for managed objects.

2. Managed vs external: the core difference

The difference comes down to who controls the physical storage layout and lifecycle. Both store data in your own Azure storage account. The table below compares them side by side.

Dimension	Managed	External
Storage path	UC chooses and controls the path under a designated managed location	You specify the exact `abfss://` path with a `LOCATION` clause
File layout and lifecycle	UC manages layout, directory structure, compaction, and cleanup	You own the file layout and lifecycle
How you reference objects	By name only (`catalog.schema.table`)	By name, and the data is also reachable by path
Access from non-Databricks tools	Through open interfaces (Iceberg REST, Delta Sharing, credential vending)	Direct path read/write at the storage path
What happens on DROP	Data is removed (8-day undrop window for tables)	Underlying files are not deleted
Automatic optimization	Yes (predictive optimization, auto liquid clustering)	No, you manage `OPTIMIZE` / `VACUUM` / statistics
Best for	Governed analytical tables and volumes created in Databricks	Fixed paths, register-in-place, sharing with external engines
Status in UC	Default and recommended	Use when a concrete path or external-tool requirement exists

3. Myth 1: "Managed means lock-in"

The word "managed" is easy to misread as "Databricks takes custody of your data" or "you can never get it back out." Neither is true.

Your data stays in your own Azure storage. Managed tables and volumes are written to an ADLS Gen2 account in your Azure subscription, governed by your network and encryption policies. Databricks does not copy your data into Databricks-owned storage. You designate the storage account; UC writes there.
"Managed" refers to file management, not ownership. It means UC handles the physical file layout, compaction, statistics, and cleanup for you. You still own the storage account, the data, and the keys.
The format is open, so there is no lock-in. Data is stored as open Delta Lake (Parquet) files, readable from outside Databricks through open interfaces: the Iceberg REST Catalog, Delta Sharing, and credential vending for engines such as Spark, Trino, DuckDB, and Snowflake. You can share it without copying.
You keep full access control. Every read and write goes through Unity Catalog privileges that you define.

A more accurate mental label for "managed" is "Databricks-optimized storage in your own account," as opposed to "Databricks-owned storage."

4. Myth 2: "Managed storage can't be segregated"

This is the one that pushes teams toward external storage unnecessarily. A managed table or volume does not all land in one big shared bucket by default. UC resolves the managed storage path in this order:

Schema managed location, if set.
Otherwise the catalog managed location, if set.
Otherwise the metastore managed location.

This gives you three levels of granularity to physically separate managed data. You can point different catalogs, or even different schemas, at completely different ADLS Gen2 containers or paths.

Metastore managed location:  abfss://metastore@examplestorage.dfs.core.windows.net/
  └── Catalog: sales          MANAGED LOCATION  abfss://sales@examplestorage.dfs.core.windows.net/
        └── Schema: orders      MANAGED LOCATION  abfss://sales-orders@examplestorage.dfs.core.windows.net/

So if the requirement is separate underlying storage per team or per sensitivity tier, managed storage meets it. You get clean physical isolation by assigning each catalog or schema its own managed location. What managed storage does not give you is control over the per-table directory names: UC creates hashed subdirectories (under __unitystorage/...) to guarantee uniqueness. You control the container/path boundary, UC controls the layout inside it.

Volumes work the same way: create separate managed volumes in separate schemas or catalogs, each with its own managed location, to segregate file storage exactly as you would with separate external paths.

One constraint to know: catalog and schema managed locations must be contained within an external location you have already created. The metastore-level managed location is the exception, and it cannot overlap any external location.

5. Managed storage: pros and cons

Pros

Automatic optimization. Predictive optimization runs OPTIMIZE, VACUUM, and ANALYZE for you. On by default for accounts created after November 11, 2024, rolling out to older accounts.
Automatic liquid clustering. UC can select and maintain clustering keys automatically.
Lower long-term cost and faster queries. Intelligent file sizing, statistics, and metadata caching reduce both storage and compute.
Less operational burden. No path management, no manual layout decisions, no orphaned-file cleanup.
Auto-upgrade to new features. Managed tables pick up new platform capabilities (catalog commits, multi-statement transactions, full-text search indexes) without re-architecting.
Strongest governance posture. No path-based access means UC privileges are always enforced. There is no side door to the files.

Cons

No direct path access. You cannot point a non-Databricks tool at the files by path. Access is through UC-governed interfaces only (which now include the Iceberg REST Catalog, Delta Sharing, and credential vending for engines like Spark, Trino, DuckDB, and Snowflake).
You do not control the directory layout. UC owns the subdirectory structure.
Drop deletes data (after the undrop window). This is a plus for hygiene, a minus if you rely on files persisting after a drop.

6. External storage: pros and cons

Pros

Explicit path control. Data lives exactly where you put it. Useful when a downstream system expects a fixed path.
Interoperability with non-Databricks tools. External tools can read and write the same files directly at the path.
Register existing data in place. Point UC at data that already exists without moving it.
Drop does not delete data. The files survive a dropped table or volume.

Cons

No automatic optimization. Predictive optimization, auto liquid clustering, and the managed-table maintenance features do not apply. You own OPTIMIZE, VACUUM, and statistics.
You own the lifecycle. Orphaned files, layout, and cleanup are your responsibility.
Governance gap on direct access. When external tools read files by path outside UC interfaces, UC privileges are not enforced on that direct file-system access. Governance is only as strong as the storage-account controls.
More moving parts. Every external path needs a storage credential, an external location, and grants. More to provision and audit.
Metadata drift. Changes made to files outside Databricks may require MSCK REPAIR TABLE ... SYNC METADATA.

7. When to choose what

Situation	Recommended model
New tables created inside Databricks, no external reader	Managed
You want automatic optimization and lowest maintenance	Managed
You need physical separation per team or sensitivity tier	Managed, with per-catalog or per-schema managed locations
Data must be read/written by a non-Databricks engine at a fixed path	External
Registering large existing datasets in place without moving them	External
Landing zones, ingestion drop folders, file-based interchange	External volume
A vendor or downstream app requires a known storage path	External
Highly regulated data where you want zero path-based side doors	Managed (no direct path access by design)

A typical multi-team environment ends up with managed as the default for governed analytical tables, and external locations reserved for ingestion landing zones, file interchange, and integration with tools that genuinely need direct path access.

8. Recommended best practices

Default to managed. Make managed tables and volumes the norm. Reach for external only when a concrete requirement (path access, external tooling, register-in-place) demands it.
Segregate with catalogs and schemas, not with external storage. Assign each team or environment its own catalog with its own MANAGED LOCATION. Use schema-level managed locations for finer isolation. This gives you physical separation while keeping the governance benefits of managed.
Keep managed and external storage in separate, dedicated containers. Managed storage cannot overlap external tables or external volumes. Plan a clean container layout up front: one set of containers for managed, separate containers for external/landing data.
One Access Connector and storage credential per trust boundary. Do not share a single credential across unrelated teams if their storage accounts have different access policies. Scope credentials to match your isolation model.
Prefer volumes over raw external-location file access. When teams need file access, govern it through a volume rather than granting READ FILES / WRITE FILES broadly on an external location.
Grant the Access Connector identity Storage Blob Data Contributor on the target ADLS Gen2 account or container. Nothing works without this role.
Use workspace bindings on external locations and credentials to restrict which workspaces can use them, if you run multiple workspaces.

9. Creating external locations on Azure

If you decide you need external storage, here is the end-to-end setup.

Permissions required

Creating external storage is a two-step, two-privilege process.

To do this	You need
Create a storage credential	`CREATE STORAGE CREDENTIAL` on the metastore. Account admins and metastore admins have it by default.
Create an external location	`CREATE EXTERNAL LOCATION` on both the metastore and the referenced storage credential. Metastore and workspace admins have it by default.

A non-admin who needs to create external locations must be granted, by a metastore admin: CREATE STORAGE CREDENTIAL on the metastore (if they will also create the credential), CREATE EXTERNAL LOCATION on the metastore, and CREATE EXTERNAL LOCATION (or MANAGE) on the specific storage credential.

Azure prerequisites

An ADLS Gen2 storage account with hierarchical namespace enabled. WORM/immutable containers are not supported.
A Databricks Access Connector for Azure with a system-assigned (or user-assigned) managed identity.
That managed identity granted the Storage Blob Data Contributor role on the storage account or container.

Step 1: Create the storage credential

On Azure, storage credentials backed by a managed identity are created through Catalog Explorer, the Databricks CLI, the REST API, or Terraform (there is no inline SQL form for Azure managed-identity credentials). The CLI form is:

databricks storage-credentials create --json '{
  "name": "adls_cred",
  "comment": "Managed identity for ADLS Gen2",
  "azure_managed_identity": {
    "access_connector_id": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Databricks/accessConnectors/<connector-name>"
  }
}'

If the Access Connector uses a user-assigned managed identity, add inside azure_managed_identity:

"managed_identity_id": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<mi-name>"

In Catalog Explorer the equivalent fields are: Credential Type = "Azure Managed Identity", "Access Connector ID", and optionally "Managed Identity ID".

Step 2: Create the external location

CREATE EXTERNAL LOCATION IF NOT EXISTS sales_landing
  URL 'abfss://landing@examplestorage.dfs.core.windows.net/sales'
  WITH (STORAGE CREDENTIAL adls_cred)
  COMMENT 'Sales landing zone';

Notes: the path is abfss://<container>@<storage-account>.dfs.core.windows.net/<path>. Names containing hyphens must be backtick-quoted. The keyword STORAGE is optional but the canonical form is WITH (STORAGE CREDENTIAL ...).

Step 3: Grant access on the external location

-- Allow a team to create external tables on this location
GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION sales_landing TO `data-engineers`;

-- Allow creating external volumes
GRANT CREATE EXTERNAL VOLUME ON EXTERNAL LOCATION sales_landing TO `data-engineers`;

-- Direct file read/write (WRITE FILES requires READ FILES too)
GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION sales_landing TO `ingestion-svc`;

-- Allow this location (or a subpath) to back a managed location for a catalog/schema
GRANT CREATE MANAGED STORAGE ON EXTERNAL LOCATION sales_landing TO `platform-admins`;

-- Review and transfer ownership
SHOW GRANTS ON EXTERNAL LOCATION sales_landing;
ALTER EXTERNAL LOCATION sales_landing OWNER TO `platform-admins`;

Defining a segregated managed location on top of external

Because catalog and schema managed locations must sit inside an external location, the pattern for segregated managed storage is: create the external location once, then point each catalog or schema's managed location at a subpath of it.

-- Catalog-level segregation: Sales managed data in its own container
CREATE CATALOG IF NOT EXISTS sales
  MANAGED LOCATION 'abfss://sales@examplestorage.dfs.core.windows.net/managed';

-- Schema-level segregation: Orders data in a separate path
CREATE SCHEMA IF NOT EXISTS sales.orders
  MANAGED LOCATION 'abfss://sales-orders@examplestorage.dfs.core.windows.net/managed';

-- A managed volume inherits the schema's managed location, physically isolated
CREATE VOLUME IF NOT EXISTS sales.orders.landing_files;

10. Hands-on walkthrough: provision it and watch the data land

Reading about resolution order is one thing. Watching a managed table land in the exact container you chose is more convincing. Here is the full sequence run end to end on an Azure workspace, with the actual output. It creates a catalog on one container and a schema on a different container, then shows that the managed table and volume follow the schema.

Azure side: storage account and access connector

RG=rg-uc-blog; LOC=eastus; SA=ucblogdemostore

az group create -n $RG -l $LOC

# ADLS Gen2 = StorageV2 with hierarchical namespace enabled
az storage account create -n $SA -g $RG -l $LOC \
  --sku Standard_LRS --kind StorageV2 --enable-hierarchical-namespace true

for c in sales sales-orders landing; do
  az storage container create --account-name $SA --name $c --auth-mode login
done

# Databricks Access Connector with a system-assigned managed identity
az databricks access-connector create -n ac-uc-blog -g $RG -l $LOC --identity-type SystemAssigned

# grant that managed identity access to the storage account
MI=$(az databricks access-connector show -n ac-uc-blog -g $RG --query identity.principalId -o tsv)
SAID=$(az storage account show -n $SA -g $RG --query id -o tsv)
az role assignment create --assignee-object-id $MI --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" --scope "$SAID"

Unity Catalog side: credential and external locations

# storage credential wrapping the access connector
databricks storage-credentials create --json '{
  "name": "adls_cred",
  "azure_managed_identity": { "access_connector_id": "<access-connector-resource-id>" }
}'

# one external location per container
databricks external-locations create sales_el        abfss://sales@ucblogdemostore.dfs.core.windows.net/        adls_cred
databricks external-locations create sales_orders_el abfss://sales-orders@ucblogdemostore.dfs.core.windows.net/ adls_cred
databricks external-locations create landing_el      abfss://landing@ucblogdemostore.dfs.core.windows.net/      adls_cred

Create the catalog and a schema on a different container

This is the segregation step: the catalog's managed location is on the sales container, but the schema declares its own managed location on sales-orders.

CREATE CATALOG av_sales_demo
  MANAGED LOCATION 'abfss://sales@ucblogdemostore.dfs.core.windows.net/managed';

CREATE SCHEMA av_sales_demo.orders
  MANAGED LOCATION 'abfss://sales-orders@ucblogdemostore.dfs.core.windows.net/managed';

CREATE TABLE av_sales_demo.orders.line_items (order_id BIGINT, region STRING, amount DECIMAL(10,2));
INSERT INTO av_sales_demo.orders.line_items VALUES (1,'EMEA',120.50),(2,'AMER',88.00),(3,'APAC',310.75);

CREATE VOLUME av_sales_demo.orders.landing_files;                                    -- managed
CREATE EXTERNAL VOLUME av_sales_demo.orders.raw_drop
  LOCATION 'abfss://landing@ucblogdemostore.dfs.core.windows.net/sales/raw_drop';    -- external

Prove where the data physically landed

DESCRIBE DETAIL  av_sales_demo.orders.line_items;
DESCRIBE VOLUME  av_sales_demo.orders.landing_files;
DESCRIBE VOLUME  av_sales_demo.orders.raw_drop;

Actual output (trimmed to the relevant fields):

-- line_items  (MANAGED table)
location: abfss://sales-orders@ucblogdemostore.../managed/__unitystorage/schemas/a412770d-.../tables/5d24725c-...

-- landing_files  (MANAGED volume)
volume_type:      MANAGED
storage_location: abfss://sales-orders@ucblogdemostore.../managed/__unitystorage/schemas/a412770d-.../volumes/d3235d73-...

-- raw_drop  (EXTERNAL volume)
volume_type:      EXTERNAL
storage_location: abfss://landing@ucblogdemostore.../sales/raw_drop

The catalog's managed location is on the sales container, yet the managed table and managed volume both landed under sales-orders, because the orders schema declared its own managed location and the schema wins over the catalog. That is physical storage segregation achieved with fully managed objects: no external tables, no loss of automatic optimization. The external volume sits exactly at the path it was given.

The same view in Catalog Explorer

Catalog Explorer: the av_sales_demo catalog with its managed storage location pointing to the sales container. Managed objects that do not declare a schema-level override will land here.

The orders schema overrides the catalog's managed location with its own, pointing at the sales-orders container. Any managed table or volume in this schema lands here instead.

The line_items managed table detail. Despite the catalog pointing at sales, the table's storage location resolves to sales-orders. UC appends hashed __unitystorage/... subdirectories; you control the container boundary.

The orders schema volumes side by side. landing_files is Managed (inherits the schema's sales-orders container). raw_drop is External (sits at the explicit abfss://landing/... path it was given).

Key takeaways

Storage credential, external location, and managed location are three distinct things. A credential is how to authenticate, an external location is which path it governs, a managed location is where managed objects land.
Managed is the default and the recommendation for governed analytical tables and volumes: automatic optimization, lower cost, strongest governance, least maintenance.
Managed does not mean lock-in. Your data stays in your own Azure storage, in open Delta format, reachable through open interfaces.
Managed storage can be physically segregated by assigning per-catalog or per-schema managed locations. The resolution order is schema > catalog > metastore.
You control the container boundary, UC controls the layout inside it. If a requirement is about the container or path boundary, managed meets it. If it is about exact directory naming or external-tool path access, that is where external fits.
External is the right tool for landing zones, file interchange, register-in-place, and non-Databricks engines. It costs you the automatic optimizations and opens a path-access governance gap, so scope it deliberately.
To create external storage you need two privileges: CREATE STORAGE CREDENTIAL for the credential, and CREATE EXTERNAL LOCATION on both the metastore and the credential for the location.
Keep managed and external in separate dedicated containers. They cannot overlap, so plan the container layout before you start.

References

Official Azure Databricks documentation:

Specify a managed storage location in Unity Catalog: metastore, catalog, and schema managed locations; resolution order.
Managed tables: default table type, automatic optimization, lifecycle.
External tables: when to use, path control, lifecycle responsibilities.
Create an external location: concepts and requirements.
Manage external locations: grants (READ FILES, WRITE FILES, CREATE EXTERNAL TABLE/VOLUME, CREATE MANAGED STORAGE).
Create a storage credential for ADLS: UI, CLI, API, and Terraform paths.
Use Azure managed identities with the Access Connector: access connector and Storage Blob Data Contributor role.
CREATE EXTERNAL LOCATION (SQL reference)
CREATE CATALOG (SQL reference)
CREATE SCHEMA (SQL reference)
Unity Catalog privileges and securable objects

Re: Managed vs External Storage in Unity Catalog on Azure: Where Should Your Data Live?

savlahanish27 — Tue, 30 Jun 2026 10:33:42 GMT

Hi Ashwin,

Really clean write-up - the schema override demo with the actual DESCRIBE DETAIL output is what makes it click. Most posts on this topic stop at "here's the theory," this one actually shows it happening.

Something similar came up on a SAP HANA to Databricks migration I worked on, with a slight twist on the landing zone side.

We had two paths feeding into Bronze - Oracle GoldenGate pushing CDC events through Kafka into Structured Streaming, and a separate JDBC job handling the historical backfill. Both needed to write to a fixed path before Databricks touched anything. GoldenGate especially - it was configured to drop files at a specific ADLS location and that path had to just exist, stable, regardless of what we were doing inside Unity Catalog. So, the landing zone ahead of Bronze ended up external, basically your "fixed path, register-in-place" case exactly.

Once Structured Streaming picked those files up and wrote them into Bronze, everything from there on was managed. Honestly that was one of the better decisions - predictive optimization and auto VACUUM just ran on their own, one less job we had to maintain. By Gold, everything was managed, partitioned by company code and fiscal year, Z-ordered on the columns that got hit hardest in joins.

So, for us it landed as: external just for that one pre-bronze hop, managed for everything after. Curious if your banking customer's CDC setup let them skip that step entirely and write straight into a managed volume, or if they hit the same fixed-path constraint we did.

topic Managed vs External Storage in Unity Catalog on Azure: Where Should Your Data Live? in Community Articles

Managed vs External Storage in Unity Catalog on Azure: Where Should Your Data Live?

1. The three building blocks

2. Managed vs external: the core difference

3. Myth 1: "Managed means lock-in"

4. Myth 2: "Managed storage can't be segregated"

5. Managed storage: pros and cons

Pros

Cons

6. External storage: pros and cons

Pros

Cons

7. When to choose what

8. Recommended best practices

9. Creating external locations on Azure

Permissions required

Azure prerequisites

Step 1: Create the storage credential

Step 2: Create the external location

Step 3: Grant access on the external location

Defining a segregated managed location on top of external

10. Hands-on walkthrough: provision it and watch the data land

Azure side: storage account and access connector

Unity Catalog side: credential and external locations

Create the catalog and a schema on a different container

Prove where the data physically landed

The same view in Catalog Explorer

Key takeaways

References

Re: Managed vs External Storage in Unity Catalog on Azure: Where Should Your Data Live?