cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

[Databricks Asset Bundles] Triggering Delta Live Tables

jonhieb
New Contributor

I would like to know how to schedule a DLT pipeline using DAB's.

I'm trying to trigger a Delta Live Table pipeline using Databricks Asset Bundles. Below is my YAML code:

resources:
  pipelines:
    data_quality_pipelines:
      name: data_quality_pipelines
trigger:
cron:
quartz_cron_schedule: "0 0 10 ? * Mon-Fri"
timezone_id: "America/Sao_Paulo"

      continuous: false          
      catalog: ${bundle.target}
      target: data_quality
      serverless: true

      libraries:
        - notebook:
            path: ../src/customfield_pipeline.ipynb
        - notebook:
            path: ../src/customfieldvalue_pipeline.ipynb
        - notebook:
            path: ../src/customer_pipeline.ipynb
        - notebook:
            path: ../src/team_pipeline.ipynb
        - notebook:
            path: ../src/user_pipeline.ipynb

      configuration:
        env_conf_file: ${var.env_conf_file}
        rules_conf_file: ${var.rules_conf_file}

After I deploy the bundle, the following error appears:

Uploading bundle files to /Workspace/Shared/deploy/.bundle/data_quality_pipelines/prod/files...
Deploying resources...
Updating deployment state...
Deployment complete!

Error: terraform apply: exit status 1

Error: cannot update pipeline: 'trigger' property is not supported yet.

with databricks_pipeline.data_quality_pipelines,
on bundle.tf.json line 61, in resource.databricks_pipeline.data_quality_pipelines:
61: }

I saw in the official documentation (Databricks API: Create Pipeline) that the trigger argument is deprecated. They recommend using the continuous argument, but I cannot configure the schedule with this command.

Does anyone know how to schedule a DLT pipeline using Databricks Asset Bundles? Should I use Databricks Workflows to orchestrate that?

 

1 ACCEPTED SOLUTION

Accepted Solutions

Walter_C
Databricks Employee
Databricks Employee

As of now, Databricks Asset Bundles do not support direct scheduling of DLT pipelines using cron expressions within the bundle configuration. Instead, you can achieve scheduling by creating a Databricks job that triggers the DLT pipeline and then scheduling the job using the Databricks Jobs API or the Databricks UI.

View solution in original post

2 REPLIES 2

Walter_C
Databricks Employee
Databricks Employee

As of now, Databricks Asset Bundles do not support direct scheduling of DLT pipelines using cron expressions within the bundle configuration. Instead, you can achieve scheduling by creating a Databricks job that triggers the DLT pipeline and then scheduling the job using the Databricks Jobs API or the Databricks UI.

That's worked for me. Thanks!!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group