- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-27-2025 01:28 PM - edited 07-27-2025 01:28 PM
Hi @sandelic ,
If you workload is mainly Databricks-centered then stick to workflows. They are easy to manage and worfklows directly integrate with Databricks notebooks and jobs.
But sometimes your workload requires complex orchestration and scheduling between many different systems and Airflow was exactly made for this. Airflow allows for extensive customization, you can author and schedule workflows programatically in Python (you can do something similar with DAB, but Airflow has more options), supports a wide range of integrations with different systems, including cloud platforms, databases, and more.
I would say, if you’re running primarily Spark-based workflows, Databricks Workflows are a great choice. However, if your data pipelines involve several different systems working together, Airflow is probably a better fit for your needs. It has steeper learning curve though.