cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Retries not triggered for a Run Job task

alejandrorr
Visitor

Is it possible to configure retries for a Databricks task of type Run Job?

I configured max_retries and min_retry_interval_millis on the task in a Databricks Asset Bundle, as I would for a notebook task. However, when the called job fails, the Run Job task does not retry. Also, the retry fields do not appear in the Jobs API get response.

Is this configuration supported for Run Job tasks? If so, how should retries be configured?

4 REPLIES 4

ashishshivatare
Databricks Partner

Hi @alejandorr

Yes, retries are supported on Run Job tasks. The Run Job task docs say to configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings, the same place youโ€™d set them for a notebook task. So the setting itself isnโ€™t the problem. databricks

The clue is that the retry fields donโ€™t show up in the Jobs API get response. That means they never reached the deployed job, so the parent task has no retry policy to apply. A few things to check:

  1. Placement in the YAML: max_retries and min_retry_interval_millis belong at the task level, as siblings of run_job_task. If theyโ€™re indented under run_job_task, they wonโ€™t be applied. It should look like this:
 
yaml
resources:
  jobs:
    parent_job:
      name: parent-job
      tasks:
        - task_key: call_child
          max_retries: 3
          min_retry_interval_millis: 60000
          retry_on_timeout: false
          run_job_task:
            job_id: ${resources.jobs.child_job.id}
  1. Check what the bundle actually sends: Run databricks bundle validate --output json and look for max_retries on that task. If itโ€™s missing there, the problem is in the YAML. Also confirm you deployed to the target youโ€™re checking in the API.
  2. Continuous trigger: If the parent job uses a continuous trigger, task retry policies donโ€™t apply. Continuous jobs use their own retry mode with exponential backoff instead.
  3. Isolate bundle vs. platform: Add retries to the Run Job task by hand in the UI, then call the get API. If the fields show up, the platform supports it and the issue is in the bundle config.

One design point to keep in mind: a retry on the Run Job task reruns the whole child job, including tasks that already succeeded. If only one task in the child job tends to fail, itโ€™s often better to put retries on that task inside the child job, so only the failing work reruns.

 

Hi  @alejandrorr,

Adding to ashishshivatare's replies: yes, the docs say retries are supported on Run Job tasks. Step 6 on the Run Job task page says "To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings." The bundle docs list max_retries, min_retry_interval_millis and retry_on_timeout under "Other task settings", which "allow you to configure behaviors for all tasks". The only task type the bundle docs say "does not support retries" is the condition task. Your 12-hour interval (43200000 ms) is within the int32 range, and the API docs list no maximum for that field, so the docs give no reason to reject it.
So the fields missing from jobs get, and the missing Retries option in the UI, are not what the docs describe. Please check two things. First, run databricks -v. If the CLI is old, update it and deploy again (this is a guess, not a documented cause). Then run databricks jobs get JOB_ID again and look at the finance task. If the fields are still missing, open a support ticket with the bundle validate JSON and the jobs get JSON that you posted here.
Until then, you can put the retry settings on the tasks of the child job (p06_pipeline_monthly_finance) itself. You can also repair the failed parent run. The repair docs say a repair re-runs "only the subset of unsuccessful tasks and any dependent tasks", and repair works only for jobs with two or more tasks. Note that a repair of the Run Job task runs the whole child job again.

Thanks, Sundar

Docs: https://docs.databricks.com/aws/en/jobs/tasks/run-job

alejandrorr
Visitor

Hi,

Thank you very much for your quick response.

I checked the bundle configuration as suggested. The retry fields appear at the task level in the output of databricks bundle validate --output json. For example, the finance task is configured as follows:

max_retries	:	10
min_retry_interval_millis	:	43200000
retry_on_timeout	:	true
	run_job_task		{2}
job_id	:	${resources.jobs.p06_pipeline_monthly_finance.id}
	job_parameters		{4}
ingestion_date	:	{{job.parameters.ingestion_date}}
reference_period_yyyymm	:	{{job.parameters.reference_period_yyyymm}}
reprocess_reason	:	{{job.parameters.reprocess_reason}}
table_suffix	:	{{job.parameters.table_suffix}}
task_key	:	finance

After running databricks bundle deploy, I can see the retry attributes in the bundle files in the Databricks workspace. However, when I call the Jobs API to retrieve the deployed job, the finance task does not include those attributes:

disabled	:	false
	email_notifications		{0}
run_if	:	ALL_SUCCESS
	run_job_task		{2}
job_id	:	513004093798668+018226-06-15T09:09:58.668Z
	job_parameters		{4}
ingestion_date	:	{{job.parameters.ingestion_date}}
reference_period_yyyymm	:	{{job.parameters.reference_period_yyyymm}}
reprocess_reason	:	{{job.parameters.reprocess_reason}}
table_suffix	:	{{job.parameters.table_suffix}}
task_key	:	finance
timeout_seconds	:	0


I also checked the task in the Databricks UI, but I cannot see an option to configure
Retries for a Run Job task. That option does appear for notebook tasks. The job is managed by a bundle and has edit_mode: UI_LOCKED. The parent job uses a scheduled trigger, not a continuous trigger.

ashishshivatare
Databricks Partner

Hi alejandrorr

Thanks for checking all of that, it rules out the bundle. bundle validate shows the retry fields at the task level, the deployed bundle files have them, and the Jobs API returns the task without them. So the backend is silently dropping retry settings on this Run Job task. The missing Retries option in the UI points the same way.

This doesnโ€™t match the Run Job task documentation, which says retries can be configured under Advanced task settings. Iโ€™d raise a support ticket with the job ID, the bundle validate output and the API get response. Either the feature isnโ€™t enabled in your workspace yet, or the docs are ahead of the platform.

Two quick checks before you open the ticket:

  1. In a scratch job that isnโ€™t bundle-managed, add a Run Job task in the UI and see whether Retries appears. If it doesnโ€™t, that confirms itโ€™s a platform limitation and not related to UI_LOCKED.
  2. Redeploy with a small min_retry_interval_millis (say 60000) and check the API again. Your current value is 12 hours. If the fields show up with a smaller value, the long interval is being rejected rather than retries being unsupported.

In the meantime, the most reliable workaround is to put the retries on the tasks inside the child job (p06_pipeline_monthly_finance) rather than on the Run Job task. Thatโ€™s often the better design anyway, because only the failing task reruns instead of the whole child pipeline:

 

resources:
  jobs:
    p06_pipeline_monthly_finance:
      tasks:
        - task_key: <task_that_fails>
          max_retries: 10
          min_retry_interval_millis: 43200000
          retry_on_timeout: true
          # existing task settings

If you need the retry at the parent level specifically, another option is to replace the Run Job task with a small notebook task that triggers the child job through the Databricks SDK (run_now_and_wait), raises an exception if the child run doesnโ€™t succeed, and carries the retry policy itself. Notebook tasks do support retries, so the parent retries the whole child run that way.