To trigger databricks workflow on defined frequency

argl1995dbks
New Contributor III

‎07-25-2024 09:33 AM

Hi Databricks, I am trying to run a databricks workflow on scheduled basis (for e.g. the frequency is after every five mins). Here is the databricks.yaml file:

 

bundle:
  name: dab_demo

# include:
#   - resources/*.yml

variables:
  job_cluster_key:
    description: Databricks job cluster
    default: job_cluster

resources:
  jobs:
    dab_demo_job:
      name: dab_demo_job
      schedule:
        quartz_cron_expression: "0 0/5 * * * ?"
        timezone_id: "Asia/Kolkata"
      #  pause_status: "PAUSED"

      email_notifications:
        on_start:
          - arjungoel1995@gmail.com
        on_success:
          - arjungoel1995@gmail.com
        on_failure:
          - arjungoel1995@gmail.com

      tasks:
        - task_key: Sum_Task
          job_cluster_key: ${var.job_cluster_key}
          spark_python_task:
            python_file: src/sum.py

        - task_key: Hello_Task
          job_cluster_key: ${var.job_cluster_key}
          depends_on:
            - task_key: Sum_Task
          spark_python_task:
            python_file: src/hello.py

      job_clusters:
        - job_cluster_key: ${var.job_cluster_key}
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: n2-highmem-4
            autoscale:
                min_workers: 2
                max_workers: 8


targets:
  dev:
    mode: development
    default: true
    workspace:
      host:<>
 
But the issue is the workflow is not getting triggered. Can you please tell if there is any issue in the schedule block and also if you can share more information about this option as I was not able to find relevant document for the same.
0 Kudos