cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

ADF vs Databricks

Phani1
Valued Contributor II

Hi Team ,

I would appreciate your suggestion on which scenario to choose between ADF (Azure Data Factory) and Databricks for orchestration, as well as any significant differences between them.

Regards,

Phanindra

2 REPLIES 2

Phani1
Valued Contributor II

Thank you for the response. Are there any significant differences between our orchestration/job scheduling methods, particularly in terms of handling databricks workflows and scheduling components other than dbt objects?

Michael_Galli
Contributor III

Hi, I work with both, so it depends on the usecase.

  • ADF is easy to set up and good for data integration, e.g. "copy data" job to transfer files from storage 1 to storage 2
  • ADF data flows (data transformations) can be used to some level, but when the transformations get more complex, I recomment to use Databricks notebooks with PySpark code
  • I am not sure how much effort Microsoft will put into ADF data flows, as in Fabric there are data flows gen 2, which are completely different to the data flows in ADF

So, for a easy low-code data ingestion and moderate data transformations I recommend ADF, and for more extensive usecases I recommend Databricks workflows.
You can combine both (Pipeline with ADF that runs a Databricks Notebook) but then you have multiple Azure services you need to take care of in terms of version control and change management.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group