AWS Databricks and Fabric OneLake integration

[de2298]

Bit of a weird scenario and I wanted to hear from the experts in this community

Lets say I have a Fabric Lakehouse (OneLake) and I want to read that data into Databricks (AWS) Unity Catalog to play with that data. What is the recommended mechanism to do this? (without actually read/writing the data)

I was looking into creating external tables in UC pointing to the blob path in OneLake, but some posts/docs out there say this is not possible. 

Any insights?

[WiliamRosa]

Hi @de2298, 

Currently, Microsoft Fabric does not offer a built-in connector that allows direct querying or exposure of Delta Share tables from AWS Databricks into a Fabric Warehouse. The Unity Catalog mirroring feature is supported only with Azure Databricks and has not been extended to AWS Databricks at this time.

- As a workaround, you can automate the ingestion of data from AWS Databricks Delta Share into an intermediate staging layer such as Azure Blob Storage or Azure SQL Database. From there, the data can be ingested into a Fabric Warehouse using T-SQL notebooks or Dataflows Gen2.
- It's also worth noting that while Lakehouse is currently the only Fabric-native destination with potential support for open Delta Sharing, Warehouse-based direct ingestion from Delta Share is not supported at this stage.
- Additionally, Fabric now supports direct access to Azure Databricks Unity Catalog tables through the Mirrored Azure Databricks Catalog feature.
- If you're working with Azure Databricks, this provides a seamless way to integrate Unity Catalog-managed data across other Fabric experiences, including Power BI.
- For AWS-hosted Delta Shares, however a middle-tier ingestion strategy remains necessary until native support becomes available.