Azure OneLake cannot be directly connected or credentialed in AWS Databricks Unity Catalog at this time, because AWS Databricks Unity Catalog supports only storage credentials for S3 and a select few options (like Cloudflare R2), rather than Azure-based offerings such as OneLake or ADLS Gen2. In contrast, the Hive Metastore is less restrictive and can be configured to access OneLake using ABFS URIs or other Azure storage connectivity features.โ
Why Unity Catalog Blocks OneLake
-
Unity Catalog in AWS currently permits storage credentials only for S3-based paths; attempts to use ABFS for OneLake or ADLS Gen2 are blocked at the permission/credential creation step.โ
-
In Azure Databricks, Unity Catalog can connect to OneLake via Azure-based identities (service principal, managed identity, or credential passthrough), but these authentication and storage mechanisms are not available in the AWS product.โ
-
There is no announced support for extending Unity Catalog's external location feature to non-S3 endpoints on AWS Databricks.โ
How Hive Metastore Differs
-
Hive Metastore does not deeply enforce cloud-native identity or storage credential constraints, so it can register and mount external tables from OneLake, provided the correct driver and endpoint details are supplied.โ
-
This flexibility lets Hive Metastore support cross-cloud sources, but comes at the cost of weaker permissions, lineage, and governance compared to Unity Catalog.โ
Workarounds and Alternatives
-
For advanced cross-cloud data scenarios, some users attempt to bridge data from OneLake to S3 (or vice versa) using integration pipelines, Delta Sharing, or direct data replication, but this is manual and typically outside Unity Catalog's governance model.โ
-
Azure Databricks with Unity Catalog offers native, seamless connectivity to OneLake, but AWS Databricks users must wait for future product support or use non-Unity Catalog patterns.โ
-
API-only approaches: In rare cases, admins may use Databricks APIs to attempt custom credential injection, but this requires privileged access and is not supported for OneLake on Unity Catalog in AWS today.โ
Summary Table
| Feature / Platform |
AWS Databricks Unity Catalog |
Azure Databricks Unity Catalog |
Hive Metastore (any cloud) |
| OneLake Connectivity |
No (S3 only) โ |
Yes (native, managed) โ |
Yes (manual) โ |
| Storage Credential Options |
S3, R2 only โ |
Azure AD, Service Principal |
Custom/driver-based |
| External Location Support |
S3 |
S3, ADLS Gen2, OneLake |
Any, via URI/driver |
For now, AWS Databricks Unity Catalog is limited to S3 locations for storage credentials and external locations, so direct integration with Azure OneLake is not possible. Using Hive Metastore remains the only practical workaround until this limitation changes or official cross-cloud support is added to Unity Catalog.โ
