Databricks Community

leelee3000 · ‎12-12-2023

I've noticed that the current development cycle for DLT jobs is quite time-consuming. The process of coding, saving, running in a workflow, and debugging seems arduous, and the feedback loop is slow. Is there a way to run DLT jobs without relying on a workflow, enabling a faster and more efficient development feedback loop? #deltalivetables

Kaniz_Fatma · ‎12-14-2023

Hi @leelee3000, Developing and iterating on Delta Live Tables (DLT) jobs can be time-consuming when relying solely on traditional workflows.

Databricks Jobs:

Databricks jobs allow you to orchestrate multiple tasks within a Databricks job, creating a data processing workflow. You can include a DLT pipeline as part of a job using the Pipeline task when completing the job.
This approach enables you to run DLT pipelines directly without the need for an external workflow. It streamlines the process by eliminating the intermediate steps of saving and running in a separate workflow.
To implement this, use the DatabricksSubmitRunOperator in Apache Airflow (an open source solution fo....

Azure Data Factory:

Azure Data Factory is a cloud-based ETL service orchestrating data integration and transformation workflows.
It directly supports running Azure Databricks tasks (including notebooks, JAR tasks, and Python scripts) within a workflow.
You can also include a DLT pipeline in a workflow by calling the Delta Live Tables API from an Azure Data Factory Web activity.
For example, you can trigger a pipeline update from Azure Data Factory by creating a new pipeline an....

Fast Iteration:

When developing DLT jobs, consider using interactive notebooks within Databricks. These notebooks allow you to rapidly write, test, and iterate on your code.
By directly executing DLT pipeline code in a notebook, you can instantly observe results, debug issues, and fine-tune your logic.
Once satisfied, you can encapsulate the tested code into a DLT pipeline and integrate it into your workflow.