ADF vs Databricks

[Phani1]

Hi Team ,

I would appreciate your suggestion on which scenario to choose between ADF (Azure Data Factory) and Databricks for orchestration, as well as any significant differences between them.

Regards,

Phanindra

[Kaniz_Fatma]

Hi @Phani1, Azure Data Factory (ADF) and Databricks are both powerful tools, but they serve different purposes and have different strengths. 

Here are some key differences:

Purpose:

• ADF is primarily used for Data Integration services to perform Extract-Transform-Load (ETL) processe....
• Databricks provides a collaborative platform for Data Engineers and Data Scientists to perform ETL a....

Ease of Usage:

• Databricks uses Python, Spark, R, Java, or SQL for performing Data Engineering and Data Science acti....
• ADF provides a drag-and-drop feature to create and maintain Data Pipelines visually. It consists of Graphical User Interface (GUI) tools that allow delivering applications at a higher r....

Flexibility in Coding:

• Although ADF facilitates the ETL pipeline process using GUI tools, developers have less flexibility ....
• Databricks implements a programmatic approach that provides the flexibility of fine-tuning codes to ....

[Phani1]

Thank you for the response. Are there any significant differences between our orchestration/job scheduling methods, particularly in terms of handling databricks workflows and scheduling components other than dbt objects?

[Michael_Galli]

Hi, I work with both, so it depends on the usecase.

• ADF is easy to set up and good for data integration, e.g. "copy data" job to transfer files from storage 1 to storage 2
• ADF data flows (data transformations) can be used to some level, but when the transformations get more complex, I recomment to use Databricks notebooks with PySpark code
• I am not sure how much effort Microsoft will put into ADF data flows, as in Fabric there are data flows gen 2, which are completely different to the data flows in ADF

So, for a easy low-code data ingestion and moderate data transformations I recommend ADF, and for more extensive usecases I recommend Databricks workflows.
You can combine both (Pipeline with ADF that runs a Databricks Notebook) but then you have multiple Azure services you need to take care of in terms of version control and change management.