I'm currently working with a banking customer migrating from Hive Metastore to Unity Catalog. While planning their catalog layout, one of their platform engineers asked the question that prompted this post: for their schemas and volumes, should they use managed or external storage, and how would they keep each business unit's data in its own storage account?
They were leaning toward external, for two reasons: they believed managed storage meant Databricks would take custody of their data, and they believed managed storage could not be physically segregated by team. I hear both on almost every Hive Metastore migration. Neither holds up. The data stays in your own Azure storage either way, and managed storage can be segregated per business unit right down to the schema. Walking through the differences gave them a clear picture of the trade-offs, and since the same question comes up so often, I figured sharing it more widely would help.
This post is that walkthrough. It covers both storage models on Azure: what they are, their trade-offs, when to pick each, where the two misconceptions break down, and a tested end-to-end example you can run yourself. Everything is backed by the Azure Databricks documentation linked at the end.
Key Takeaways
- "Managed" means Databricks optimizes your files. It does not mean Databricks owns your data. Your data stays in your own Azure storage, in open Delta format.
- Managed storage can be physically segregated per team or environment by assigning per-catalog or per-schema managed locations. The resolution order is schema > catalog > metastore.
- Managed is the default recommendation for governed analytical tables and volumes. External is the right tool for landing zones, fixed paths, register-in-place, and non-Databricks engines.
- Creating external storage requires two separate privileges: CREATE STORAGE CREDENTIAL on the metastore, and CREATE EXTERNAL LOCATION on both the metastore and the credential.
What's in this post
- The three building blocks
- Managed vs external: the core difference
- Myth 1: "Managed means lock-in"
- Myth 2: "Managed storage can't be segregated"
- Managed storage: pros and cons
- External storage: pros and cons
- When to choose what
- Recommended best practices
- Creating external locations on Azure
- Hands-on walkthrough: provision it and watch the data land
- References
1. The three building blocks
Before comparing the two models, it helps to separate three concepts that are often conflated.
| Concept | What it is | Scope |
|---|---|---|
| Storage credential | A securable that holds the authentication to your Azure storage. On Azure this is an Azure Managed Identity attached to a Databricks Access Connector. | Metastore |
| External location | A securable that pairs a storage credential with a specific abfss:// path. It is the governed gateway to a path in ADLS Gen2. |
Metastore |
| Managed location | A path (which must sit inside an external location) where UC stores managed tables and volumes. Can be set at metastore, catalog, or schema level. | Metastore / Catalog / Schema |
The mental model: a storage credential says how to authenticate, an external location says which path that credential governs, and a managed location designates a path where UC takes over file layout for managed objects.
2. Managed vs external: the core difference
The difference comes down to who controls the physical storage layout and lifecycle. Both store data in your own Azure storage account. The table below compares them side by side.
| Dimension | Managed | External |
|---|---|---|
| Storage path | UC chooses and controls the path under a designated managed location | You specify the exact abfss:// path with a LOCATION clause |
| File layout and lifecycle | UC manages layout, directory structure, compaction, and cleanup | You own the file layout and lifecycle |
| How you reference objects | By name only (catalog.schema.table) |
By name, and the data is also reachable by path |
| Access from non-Databricks tools | Through open interfaces (Iceberg REST, Delta Sharing, credential vending) | Direct path read/write at the storage path |
| What happens on DROP | Data is removed (8-day undrop window for tables) | Underlying files are not deleted |
| Automatic optimization | Yes (predictive optimization, auto liquid clustering) | No, you manage OPTIMIZE / VACUUM / statistics |
| Best for | Governed analytical tables and volumes created in Databricks | Fixed paths, register-in-place, sharing with external engines |
| Status in UC | Default and recommended | Use when a concrete path or external-tool requirement exists |
3. Myth 1: "Managed means lock-in"
The word "managed" is easy to misread as "Databricks takes custody of your data" or "you can never get it back out." Neither is true.
- Your data stays in your own Azure storage. Managed tables and volumes are written to an ADLS Gen2 account in your Azure subscription, governed by your network and encryption policies. Databricks does not copy your data into Databricks-owned storage. You designate the storage account; UC writes there.
- "Managed" refers to file management, not ownership. It means UC handles the physical file layout, compaction, statistics, and cleanup for you. You still own the storage account, the data, and the keys.
- The format is open, so there is no lock-in. Data is stored as open Delta Lake (Parquet) files, readable from outside Databricks through open interfaces: the Iceberg REST Catalog, Delta Sharing, and credential vending for engines such as Spark, Trino, DuckDB, and Snowflake. You can share it without copying.
- You keep full access control. Every read and write goes through Unity Catalog privileges that you define.
A more accurate mental label for "managed" is "Databricks-optimized storage in your own account," as opposed to "Databricks-owned storage."
4. Myth 2: "Managed storage can't be segregated"
This is the one that pushes teams toward external storage unnecessarily. A managed table or volume does not all land in one big shared bucket by default. UC resolves the managed storage path in this order:
- Schema managed location, if set.
- Otherwise the catalog managed location, if set.
- Otherwise the metastore managed location.
This gives you three levels of granularity to physically separate managed data. You can point different catalogs, or even different schemas, at completely different ADLS Gen2 containers or paths.
Metastore managed location: abfss://metastore@examplestorage.dfs.core.windows.net/
└── Catalog: sales MANAGED LOCATION abfss://sales@examplestorage.dfs.core.windows.net/
└── Schema: orders MANAGED LOCATION abfss://sales-orders@examplestorage.dfs.core.windows.net/
So if the requirement is separate underlying storage per team or per sensitivity tier, managed storage meets it. You get clean physical isolation by assigning each catalog or schema its own managed location. What managed storage does not give you is control over the per-table directory names: UC creates hashed subdirectories (under __unitystorage/...) to guarantee uniqueness. You control the container/path boundary, UC controls the layout inside it.
Volumes work the same way: create separate managed volumes in separate schemas or catalogs, each with its own managed location, to segregate file storage exactly as you would with separate external paths.
One constraint to know: catalog and schema managed locations must be contained within an external location you have already created. The metastore-level managed location is the exception, and it cannot overlap any external location.
5. Managed storage: pros and cons
Pros
- Automatic optimization. Predictive optimization runs
OPTIMIZE,VACUUM, andANALYZEfor you. On by default for accounts created after November 11, 2024, rolling out to older accounts. - Automatic liquid clustering. UC can select and maintain clustering keys automatically.
- Lower long-term cost and faster queries. Intelligent file sizing, statistics, and metadata caching reduce both storage and compute.
- Less operational burden. No path management, no manual layout decisions, no orphaned-file cleanup.
- Auto-upgrade to new features. Managed tables pick up new platform capabilities (catalog commits, multi-statement transactions, full-text search indexes) without re-architecting.
- Strongest governance posture. No path-based access means UC privileges are always enforced. There is no side door to the files.
Cons
- No direct path access. You cannot point a non-Databricks tool at the files by path. Access is through UC-governed interfaces only (which now include the Iceberg REST Catalog, Delta Sharing, and credential vending for engines like Spark, Trino, DuckDB, and Snowflake).
- You do not control the directory layout. UC owns the subdirectory structure.
- Drop deletes data (after the undrop window). This is a plus for hygiene, a minus if you rely on files persisting after a drop.
6. External storage: pros and cons
Pros
- Explicit path control. Data lives exactly where you put it. Useful when a downstream system expects a fixed path.
- Interoperability with non-Databricks tools. External tools can read and write the same files directly at the path.
- Register existing data in place. Point UC at data that already exists without moving it.
- Drop does not delete data. The files survive a dropped table or volume.
Cons
- No automatic optimization. Predictive optimization, auto liquid clustering, and the managed-table maintenance features do not apply. You own
OPTIMIZE,VACUUM, and statistics. - You own the lifecycle. Orphaned files, layout, and cleanup are your responsibility.
- Governance gap on direct access. When external tools read files by path outside UC interfaces, UC privileges are not enforced on that direct file-system access. Governance is only as strong as the storage-account controls.
- More moving parts. Every external path needs a storage credential, an external location, and grants. More to provision and audit.
- Metadata drift. Changes made to files outside Databricks may require
MSCK REPAIR TABLE ... SYNC METADATA.
7. When to choose what
| Situation | Recommended model |
|---|---|
| New tables created inside Databricks, no external reader | Managed |
| You want automatic optimization and lowest maintenance | Managed |
| You need physical separation per team or sensitivity tier | Managed, with per-catalog or per-schema managed locations |
| Data must be read/written by a non-Databricks engine at a fixed path | External |
| Registering large existing datasets in place without moving them | External |
| Landing zones, ingestion drop folders, file-based interchange | External volume |
| A vendor or downstream app requires a known storage path | External |
| Highly regulated data where you want zero path-based side doors | Managed (no direct path access by design) |
A typical multi-team environment ends up with managed as the default for governed analytical tables, and external locations reserved for ingestion landing zones, file interchange, and integration with tools that genuinely need direct path access.
8. Recommended best practices
- Default to managed. Make managed tables and volumes the norm. Reach for external only when a concrete requirement (path access, external tooling, register-in-place) demands it.
- Segregate with catalogs and schemas, not with external storage. Assign each team or environment its own catalog with its own
MANAGED LOCATION. Use schema-level managed locations for finer isolation. This gives you physical separation while keeping the governance benefits of managed. - Keep managed and external storage in separate, dedicated containers. Managed storage cannot overlap external tables or external volumes. Plan a clean container layout up front: one set of containers for managed, separate containers for external/landing data.
- One Access Connector and storage credential per trust boundary. Do not share a single credential across unrelated teams if their storage accounts have different access policies. Scope credentials to match your isolation model.
- Prefer volumes over raw external-location file access. When teams need file access, govern it through a volume rather than granting
READ FILES/WRITE FILESbroadly on an external location. - Grant the Access Connector identity Storage Blob Data Contributor on the target ADLS Gen2 account or container. Nothing works without this role.
- Use workspace bindings on external locations and credentials to restrict which workspaces can use them, if you run multiple workspaces.
9. Creating external locations on Azure
If you decide you need external storage, here is the end-to-end setup.
Permissions required
Creating external storage is a two-step, two-privilege process.
| To do this | You need |
|---|---|
| Create a storage credential | CREATE STORAGE CREDENTIAL on the metastore. Account admins and metastore admins have it by default. |
| Create an external location | CREATE EXTERNAL LOCATION on both the metastore and the referenced storage credential. Metastore and workspace admins have it by default. |
A non-admin who needs to create external locations must be granted, by a metastore admin: CREATE STORAGE CREDENTIAL on the metastore (if they will also create the credential), CREATE EXTERNAL LOCATION on the metastore, and CREATE EXTERNAL LOCATION (or MANAGE) on the specific storage credential.
Azure prerequisites
- An ADLS Gen2 storage account with hierarchical namespace enabled. WORM/immutable containers are not supported.
- A Databricks Access Connector for Azure with a system-assigned (or user-assigned) managed identity.
- That managed identity granted the Storage Blob Data Contributor role on the storage account or container.
Step 1: Create the storage credential
On Azure, storage credentials backed by a managed identity are created through Catalog Explorer, the Databricks CLI, the REST API, or Terraform (there is no inline SQL form for Azure managed-identity credentials). The CLI form is:
databricks storage-credentials create --json '{
"name": "adls_cred",
"comment": "Managed identity for ADLS Gen2",
"azure_managed_identity": {
"access_connector_id": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.Databricks/accessConnectors/<connector-name>"
}
}'
If the Access Connector uses a user-assigned managed identity, add inside azure_managed_identity:
"managed_identity_id": "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<mi-name>"
In Catalog Explorer the equivalent fields are: Credential Type = "Azure Managed Identity", "Access Connector ID", and optionally "Managed Identity ID".
Step 2: Create the external location
CREATE EXTERNAL LOCATION IF NOT EXISTS sales_landing URL 'abfss://landing@examplestorage.dfs.core.windows.net/sales' WITH (STORAGE CREDENTIAL adls_cred) COMMENT 'Sales landing zone';
Notes: the path is abfss://<container>@<storage-account>.dfs.core.windows.net/<path>. Names containing hyphens must be backtick-quoted. The keyword STORAGE is optional but the canonical form is WITH (STORAGE CREDENTIAL ...).
Step 3: Grant access on the external location
-- Allow a team to create external tables on this location GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION sales_landing TO `data-engineers`; -- Allow creating external volumes GRANT CREATE EXTERNAL VOLUME ON EXTERNAL LOCATION sales_landing TO `data-engineers`; -- Direct file read/write (WRITE FILES requires READ FILES too) GRANT READ FILES, WRITE FILES ON EXTERNAL LOCATION sales_landing TO `ingestion-svc`; -- Allow this location (or a subpath) to back a managed location for a catalog/schema GRANT CREATE MANAGED STORAGE ON EXTERNAL LOCATION sales_landing TO `platform-admins`; -- Review and transfer ownership SHOW GRANTS ON EXTERNAL LOCATION sales_landing; ALTER EXTERNAL LOCATION sales_landing OWNER TO `platform-admins`;
Defining a segregated managed location on top of external
Because catalog and schema managed locations must sit inside an external location, the pattern for segregated managed storage is: create the external location once, then point each catalog or schema's managed location at a subpath of it.
-- Catalog-level segregation: Sales managed data in its own container CREATE CATALOG IF NOT EXISTS sales MANAGED LOCATION 'abfss://sales@examplestorage.dfs.core.windows.net/managed'; -- Schema-level segregation: Orders data in a separate path CREATE SCHEMA IF NOT EXISTS sales.orders MANAGED LOCATION 'abfss://sales-orders@examplestorage.dfs.core.windows.net/managed'; -- A managed volume inherits the schema's managed location, physically isolated CREATE VOLUME IF NOT EXISTS sales.orders.landing_files;
10. Hands-on walkthrough: provision it and watch the data land
Reading about resolution order is one thing. Watching a managed table land in the exact container you chose is more convincing. Here is the full sequence run end to end on an Azure workspace, with the actual output. It creates a catalog on one container and a schema on a different container, then shows that the managed table and volume follow the schema.
Azure side: storage account and access connector
RG=rg-uc-blog; LOC=eastus; SA=ucblogdemostore az group create -n $RG -l $LOC # ADLS Gen2 = StorageV2 with hierarchical namespace enabled az storage account create -n $SA -g $RG -l $LOC \ --sku Standard_LRS --kind StorageV2 --enable-hierarchical-namespace true for c in sales sales-orders landing; do az storage container create --account-name $SA --name $c --auth-mode login done # Databricks Access Connector with a system-assigned managed identity az databricks access-connector create -n ac-uc-blog -g $RG -l $LOC --identity-type SystemAssigned # grant that managed identity access to the storage account MI=$(az databricks access-connector show -n ac-uc-blog -g $RG --query identity.principalId -o tsv) SAID=$(az storage account show -n $SA -g $RG --query id -o tsv) az role assignment create --assignee-object-id $MI --assignee-principal-type ServicePrincipal \ --role "Storage Blob Data Contributor" --scope "$SAID"
Unity Catalog side: credential and external locations
# storage credential wrapping the access connector
databricks storage-credentials create --json '{
"name": "adls_cred",
"azure_managed_identity": { "access_connector_id": "<access-connector-resource-id>" }
}'
# one external location per container
databricks external-locations create sales_el abfss://sales@ucblogdemostore.dfs.core.windows.net/ adls_cred
databricks external-locations create sales_orders_el abfss://sales-orders@ucblogdemostore.dfs.core.windows.net/ adls_cred
databricks external-locations create landing_el abfss://landing@ucblogdemostore.dfs.core.windows.net/ adls_cred
Create the catalog and a schema on a different container
This is the segregation step: the catalog's managed location is on the sales container, but the schema declares its own managed location on sales-orders.
CREATE CATALOG av_sales_demo MANAGED LOCATION 'abfss://sales@ucblogdemostore.dfs.core.windows.net/managed'; CREATE SCHEMA av_sales_demo.orders MANAGED LOCATION 'abfss://sales-orders@ucblogdemostore.dfs.core.windows.net/managed'; CREATE TABLE av_sales_demo.orders.line_items (order_id BIGINT, region STRING, amount DECIMAL(10,2)); INSERT INTO av_sales_demo.orders.line_items VALUES (1,'EMEA',120.50),(2,'AMER',88.00),(3,'APAC',310.75); CREATE VOLUME av_sales_demo.orders.landing_files; -- managed CREATE EXTERNAL VOLUME av_sales_demo.orders.raw_drop LOCATION 'abfss://landing@ucblogdemostore.dfs.core.windows.net/sales/raw_drop'; -- external
Prove where the data physically landed
DESCRIBE DETAIL av_sales_demo.orders.line_items; DESCRIBE VOLUME av_sales_demo.orders.landing_files; DESCRIBE VOLUME av_sales_demo.orders.raw_drop;
Actual output (trimmed to the relevant fields):
-- line_items (MANAGED table) location: abfss://sales-orders@ucblogdemostore.../managed/__unitystorage/schemas/a412770d-.../tables/5d24725c-... -- landing_files (MANAGED volume) volume_type: MANAGED storage_location: abfss://sales-orders@ucblogdemostore.../managed/__unitystorage/schemas/a412770d-.../volumes/d3235d73-... -- raw_drop (EXTERNAL volume) volume_type: EXTERNAL storage_location: abfss://landing@ucblogdemostore.../sales/raw_drop
The catalog's managed location is on the sales container, yet the managed table and managed volume both landed under sales-orders, because the orders schema declared its own managed location and the schema wins over the catalog. That is physical storage segregation achieved with fully managed objects: no external tables, no loss of automatic optimization. The external volume sits exactly at the path it was given.
The same view in Catalog Explorer
Catalog Explorer: the av_sales_demo catalog with its managed storage location pointing to the sales container. Managed objects that do not declare a schema-level override will land here.
The orders schema overrides the catalog's managed location with its own, pointing at the sales-orders container. Any managed table or volume in this schema lands here instead.
The line_items managed table detail. Despite the catalog pointing at sales, the table's storage location resolves to sales-orders. UC appends hashed __unitystorage/... subdirectories; you control the container boundary.
The orders schema volumes side by side. landing_files is Managed (inherits the schema's sales-orders container). raw_drop is External (sits at the explicit abfss://landing/... path it was given).Key takeaways
- Storage credential, external location, and managed location are three distinct things. A credential is how to authenticate, an external location is which path it governs, a managed location is where managed objects land.
- Managed is the default and the recommendation for governed analytical tables and volumes: automatic optimization, lower cost, strongest governance, least maintenance.
- Managed does not mean lock-in. Your data stays in your own Azure storage, in open Delta format, reachable through open interfaces.
- Managed storage can be physically segregated by assigning per-catalog or per-schema managed locations. The resolution order is schema > catalog > metastore.
- You control the container boundary, UC controls the layout inside it. If a requirement is about the container or path boundary, managed meets it. If it is about exact directory naming or external-tool path access, that is where external fits.
- External is the right tool for landing zones, file interchange, register-in-place, and non-Databricks engines. It costs you the automatic optimizations and opens a path-access governance gap, so scope it deliberately.
- To create external storage you need two privileges:
CREATE STORAGE CREDENTIALfor the credential, andCREATE EXTERNAL LOCATIONon both the metastore and the credential for the location. - Keep managed and external in separate dedicated containers. They cannot overlap, so plan the container layout before you start.
References
Official Azure Databricks documentation:
- Specify a managed storage location in Unity Catalog: metastore, catalog, and schema managed locations; resolution order.
- Managed tables: default table type, automatic optimization, lifecycle.
- External tables: when to use, path control, lifecycle responsibilities.
- Create an external location: concepts and requirements.
- Manage external locations: grants (READ FILES, WRITE FILES, CREATE EXTERNAL TABLE/VOLUME, CREATE MANAGED STORAGE).
- Create a storage credential for ADLS: UI, CLI, API, and Terraform paths.
- Use Azure managed identities with the Access Connector: access connector and Storage Blob Data Contributor role.
- CREATE EXTERNAL LOCATION (SQL reference)
- CREATE CATALOG (SQL reference)
- CREATE SCHEMA (SQL reference)
- Unity Catalog privileges and securable objects
Announcements
