topic Re: Fabric with Databricks in Warehousing & Analytics

Fabric with Databricks

DEShoaib — Sat, 01 Feb 2025 10:12:04 GMT

Do we have same functionality if we use Databricks with Fabric as it provides with Azure?

Re: Fabric with Databricks

Alberto_Umana — Sat, 01 Feb 2025 16:02:41 GMT

Are you looking for any functionality in specific?

Some of the limitations are:

Governance and Security: Fabric does not enforce Unity Catalog security policies on downstream users, and it does not support fine-grained access control, views, materialized views, streaming tables, or tables with row-level filters or column masks.
Data Copying: Some integrations involve copying data into OneLake, which can introduce additional costs and governance challenges.
Performance and Cost: Using OneLake and Direct Lake can be more expensive due to the need for a running Fabric capacity and the 3X cost penalty for accessing data from non-Fabric tools

Re: Fabric with Databricks

DEShoaib — Mon, 03 Feb 2025 14:55:06 GMT

Actually leadership is asking me to evaluate the possibility if we can only use Fabric and migrate our existing workload from Azure Synapse to Fabric Datawarehouse

Re: Fabric with Databricks

Mantsama4 — Mon, 03 Feb 2025 15:56:47 GMT

While both Microsoft Fabric and Databricks provide advanced data analytics capabilities, their functionalities differ significantly based on use cases, technical complexity, and cloud flexibility.

1️⃣ Architecture & Integration

Fabric is a fully integrated Azure ecosystem platform, combining OneLake, Synapse, Data Factory, and Power BI for seamless, low-code data operations.
Databricks, on the other hand, operates as a multi-cloud, high-performance lakehouse, leveraging Apache Spark, Delta Lake, and MLflow for advanced data engineering, AI/ML, and scalable analytics.

2️⃣ Ease of Use vs. Advanced Capabilities

Fabric is designed for business analysts and citizen data scientists, with a user-friendly, low-code/no-code experience.
Databricks is built for data engineers and data scientists, requiring more technical expertise but providing deep customization and scalability.

3️⃣ Cloud Flexibility

Fabric is tightly integrated with Azure, ideal for Microsoft-first enterprises.
Databricks operates across Azure, AWS, and GCP, offering multi-cloud flexibility.

4️⃣ Data Science & AI Capabilities

Fabric has limited AI/ML capabilities, suitable for simpler analytics tasks.
Databricks offers best-in-class AI/ML tools, including MLflow, Feature Store, and Databricks Model Serving, enabling scalable ML workflows.

5️⃣ Pricing Model

Fabric uses a capacity-based pricing model, bundling compute, storage, and data transfer into fixed tiers.
Databricks follows a pay-as-you-go consumption model, allowing granular cost control and optimization.

Final Takeaway:
If your organization prioritizes deep AI/ML capabilities, multi-cloud flexibility, and large-scale data engineering, Databricks is the stronger choice. However, if you require seamless Azure integration, real-time analytics, and a low-code experience, Fabric may be a better fit. The choice depends on your use case, team expertise, and cloud strategy.

Re: Fabric with Databricks

MariuszK — Mon, 03 Feb 2025 15:59:15 GMT

hi @DEShoaib
Are you planing to move Dedicated (T-SQL) pool or Spark code?
With Databricks you can replicated all features from Azure Synapse, you have possibility to use PySpark and Databricks SQL. MS Fabric has nice integration with Power BI and easy configuration. You can also use Databricks with MS Fabric using Databricks Mirroring so you will have access to all tables available in UC, but Mirroring doesn't replicate security configuration from UC.

You can read more about mirroring and integrations here: MS Fabric Databricks Mirroring

Re: Fabric with Databricks

DEShoaib — Mon, 03 Feb 2025 16:05:52 GMT

We are planning to migrate both T-SQL workloads from the Dedicated SQL Pool and some Spark code where we are consuming data from APIs. Our goal is to evaluate if Microsoft Fabric can fully replace our existing Synapse environment while ensuring smooth integration with Power BI.

Since Fabric Data Warehouse supports T-SQL and Spark Notebooks are available within Synapse Data Engineering, we are exploring whether Fabric can handle our Spark-based API ingestion as well. Additionally, we are considering Databricks Mirroring as an option to integrate with Fabric, but we need to assess the impact of security configurations not being replicated from Unity Catalog.

Re: Fabric with Databricks

MariuszK — Mon, 03 Feb 2025 16:22:54 GMT

It make sense to move T-SQL to an engine supporting it such as Fabric Warehouse, but I think this kind of migration requires dipper evaluation in the context of future costs and possibilities. MS Fabric is still new. Databricks is a more mature solution, but in your case, you would need to translate T-SQL to Spark/SparkSQL (it's possible to simplify it). Databricks has integration with Power BI and it works well with SQL Warehouse.

I think it's not just mater of moving, but analyzing it in wider perspective: cost, features, team capabilities, etc.

Re: Fabric with Databricks

dks — Mon, 19 May 2025 08:03:14 GMT

We aim to implement Databricks Mirroring through the Fabric APIs for automation. However, the Mirroring API specifically states that it is not compatible with Databricks. Are there alternative APIs that could be used to achieve this functionality?