cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

[Delta live table vs Workflow]

NhanNguyen
Contributor II

Hi Community Members,

I have been using Databricks for a while, but I have only used Workflow. I have a question about the differences between Delta Live Table and Workflow. Which one should we use in which scenario?

Thanks,

2 REPLIES 2

Hkesharwani
Contributor II

Hi, 

  1. Delta Live Tables focuses on managing data ingestion, transformation, and management of Delta tables using a declarative framework. Job Workflows are designed to orchestrate and schedule various data processing and analysis tasks, including SQL queries, machine learning jobs, and notebook execution.
    Data Processing Paradigm.
  2. Both of them support batch workload and streaming workload, but delta supports almost real-time transformation. 
  3. Delta Live Tables abstracts away many of the complexities of traditional ETL and streaming data processing pipelines. It handles task orchestration, cluster management, monitoring, and error handling.
  4. Job Workflows provide a broader level of control by allowing you to define and manage complex workflows consisting of multiple tasks or jobs. It provides control over dependencies, resource allocation, and monitoring of job execution.
  5. Delta Live Tables leverages Delta Lake as the underlying storage engine for data management, providing features like schema evolution, ACID transactions, and data versioning. It seamlessly integrates with Delta Lake APIs and functionalities. Job Workflows can also interact with Delta tables and leverage the capabilities of Delta Lake. However, they are not specifically designed to address the complexities of working with Delta tables like Delta Live Tables.
  6. One of the best feature of delta live tables is live quality control monitoring.

Check these article from datarbricks to understand more about the delta live tables. 

https://community.databricks.com/t5/community-discussions/delta-live-table-vs-workflow/td-p/69373 
https://www.databricks.com/blog/2023/04/14/how-we-performed-etl-one-billion-records-under-1-delta-li...

Harshit Kesharwani
Self-taught Data Engineer | Seeking Remote Full-time Opportunities

Hi @Hkesharwani,

Thanks for your answer. This information is very detailed and valuable to me!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!