cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Retries not triggered for a Run Job task

alejandrorr
Visitor

Is it possible to configure retries for a Databricks task of type Run Job?

I configured max_retries and min_retry_interval_millis on the task in a Databricks Asset Bundle, as I would for a notebook task. However, when the called job fails, the Run Job task does not retry. Also, the retry fields do not appear in the Jobs API get response.

Is this configuration supported for Run Job tasks? If so, how should retries be configured?

1 REPLY 1

ashishshivatare
Databricks Partner

Hi @alejandorr

Yes, retries are supported on Run Job tasks. The Run Job task docs say to configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings, the same place youโ€™d set them for a notebook task. So the setting itself isnโ€™t the problem. databricks

The clue is that the retry fields donโ€™t show up in the Jobs API get response. That means they never reached the deployed job, so the parent task has no retry policy to apply. A few things to check:

  1. Placement in the YAML: max_retries and min_retry_interval_millis belong at the task level, as siblings of run_job_task. If theyโ€™re indented under run_job_task, they wonโ€™t be applied. It should look like this:
 
yaml
resources:
  jobs:
    parent_job:
      name: parent-job
      tasks:
        - task_key: call_child
          max_retries: 3
          min_retry_interval_millis: 60000
          retry_on_timeout: false
          run_job_task:
            job_id: ${resources.jobs.child_job.id}
  1. Check what the bundle actually sends: Run databricks bundle validate --output json and look for max_retries on that task. If itโ€™s missing there, the problem is in the YAML. Also confirm you deployed to the target youโ€™re checking in the API.
  2. Continuous trigger: If the parent job uses a continuous trigger, task retry policies donโ€™t apply. Continuous jobs use their own retry mode with exponential backoff instead.
  3. Isolate bundle vs. platform: Add retries to the Run Job task by hand in the UI, then call the get API. If the fields show up, the platform supports it and the issue is in the bundle config.

One design point to keep in mind: a retry on the Run Job task reruns the whole child job, including tasks that already succeeded. If only one task in the child job tends to fail, itโ€™s often better to put retries on that task inside the child job, so only the failing work reruns.