davidmorton
Databricks Employee
Databricks Employee

Unfortunately, you can't set it in YAML directly. 

A workaround for this would be to create the pipeline as a triggered pipeline, and then create a job that will trigger the pipeline periodically. Unlike pipelines, job schedules can be defined using the YAML. 

Something like the following would probably get the job done:

# resources/my_pipeline_job.yml
resources:
  jobs:
    scheduled_dlt_pipeline:
      name: "Scheduled DLT Pipeline Job"
      schedule:
        quartz_cron_expression: "0 0 8 * * ?"  # Daily at 8 AM
        timezone_id: "America/Chicago"
        pause_status: "UNPAUSED"
      tasks:
        - task_key: run_dlt_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.my_dlt_pipeline.id}
      email_notifications:
        on_failure:
          - data-team@company.com