how to use azure one lake in aws databricks unity catalog

ashish31negi — Mon, 24 Mar 2025 16:24:03 GMT

i'm trying to connect azure one lake in aws databricks unity catalog but i'm not able to storage credential, since it's currently allowing s3 location only but in hive catalog i'm able to connect to one lake but not in unity.

Re: how to use azure one lake in aws databricks unity catalog

mark_ott — Wed, 29 Oct 2025 20:40:57 GMT

Azure OneLake cannot be directly connected or credentialed in AWS Databricks Unity Catalog at this time, because AWS Databricks Unity Catalog supports only storage credentials for S3 and a select few options (like Cloudflare R2), rather than Azure-based offerings such as OneLake or ADLS Gen2. In contrast, the Hive Metastore is less restrictive and can be configured to access OneLake using ABFS URIs or other Azure storage connectivity features.

Why Unity Catalog Blocks OneLake

Unity Catalog in AWS currently permits storage credentials only for S3-based paths; attempts to use ABFS for OneLake or ADLS Gen2 are blocked at the permission/credential creation step.
In Azure Databricks, Unity Catalog can connect to OneLake via Azure-based identities (service principal, managed identity, or credential passthrough), but these authentication and storage mechanisms are not available in the AWS product.
There is no announced support for extending Unity Catalog's external location feature to non-S3 endpoints on AWS Databricks.

How Hive Metastore Differs

Hive Metastore does not deeply enforce cloud-native identity or storage credential constraints, so it can register and mount external tables from OneLake, provided the correct driver and endpoint details are supplied.
This flexibility lets Hive Metastore support cross-cloud sources, but comes at the cost of weaker permissions, lineage, and governance compared to Unity Catalog.

Workarounds and Alternatives

For advanced cross-cloud data scenarios, some users attempt to bridge data from OneLake to S3 (or vice versa) using integration pipelines, Delta Sharing, or direct data replication, but this is manual and typically outside Unity Catalog's governance model.
Azure Databricks with Unity Catalog offers native, seamless connectivity to OneLake, but AWS Databricks users must wait for future product support or use non-Unity Catalog patterns.
API-only approaches: In rare cases, admins may use Databricks APIs to attempt custom credential injection, but this requires privileged access and is not supported for OneLake on Unity Catalog in AWS today.

Summary Table

Feature / Platform	AWS Databricks Unity Catalog	Azure Databricks Unity Catalog	Hive Metastore (any cloud)
OneLake Connectivity	No (S3 only)	Yes (native, managed)	Yes (manual)
Storage Credential Options	S3, R2 only	Azure AD, Service Principal	Custom/driver-based
External Location Support	S3	S3, ADLS Gen2, OneLake	Any, via URI/driver