19 hours ago
Is it possible to configure retries for a Databricks task of type Run Job?
I configured max_retries and min_retry_interval_millis on the task in a Databricks Asset Bundle, as I would for a notebook task. However, when the called job fails, the Run Job task does not retry. Also, the retry fields do not appear in the Jobs API get response.
Is this configuration supported for Run Job tasks? If so, how should retries be configured?
18 hours ago
Hi @alejandorr
Yes, retries are supported on Run Job tasks. The Run Job task docs say to configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings, the same place youโd set them for a notebook task. So the setting itself isnโt the problem. databricks
The clue is that the retry fields donโt show up in the Jobs API get response. That means they never reached the deployed job, so the parent task has no retry policy to apply. A few things to check:
resources:
jobs:
parent_job:
name: parent-job
tasks:
- task_key: call_child
max_retries: 3
min_retry_interval_millis: 60000
retry_on_timeout: false
run_job_task:
job_id: ${resources.jobs.child_job.id}One design point to keep in mind: a retry on the Run Job task reruns the whole child job, including tasks that already succeeded. If only one task in the child job tends to fail, itโs often better to put retries on that task inside the child job, so only the failing work reruns.
2 hours ago
Hi @alejandrorr,
Adding to ashishshivatare's replies: yes, the docs say retries are supported on Run Job tasks. Step 6 on the Run Job task page says "To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings." The bundle docs list max_retries, min_retry_interval_millis and retry_on_timeout under "Other task settings", which "allow you to configure behaviors for all tasks". The only task type the bundle docs say "does not support retries" is the condition task. Your 12-hour interval (43200000 ms) is within the int32 range, and the API docs list no maximum for that field, so the docs give no reason to reject it.
So the fields missing from jobs get, and the missing Retries option in the UI, are not what the docs describe. Please check two things. First, run databricks -v. If the CLI is old, update it and deploy again (this is a guess, not a documented cause). Then run databricks jobs get JOB_ID again and look at the finance task. If the fields are still missing, open a support ticket with the bundle validate JSON and the jobs get JSON that you posted here.
Until then, you can put the retry settings on the tasks of the child job (p06_pipeline_monthly_finance) itself. You can also repair the failed parent run. The repair docs say a repair re-runs "only the subset of unsuccessful tasks and any dependent tasks", and repair works only for jobs with two or more tasks. Note that a repair of the Run Job task runs the whole child job again.
Thanks, Sundar
17 hours ago
Hi,
Thank you very much for your quick response.
I checked the bundle configuration as suggested. The retry fields appear at the task level in the output of databricks bundle validate --output json. For example, the finance task is configured as follows:
max_retries : 10
min_retry_interval_millis : 43200000
retry_on_timeout : true
run_job_task {2}
job_id : ${resources.jobs.p06_pipeline_monthly_finance.id}
job_parameters {4}
ingestion_date : {{job.parameters.ingestion_date}}
reference_period_yyyymm : {{job.parameters.reference_period_yyyymm}}
reprocess_reason : {{job.parameters.reprocess_reason}}
table_suffix : {{job.parameters.table_suffix}}
task_key : financeAfter running databricks bundle deploy, I can see the retry attributes in the bundle files in the Databricks workspace. However, when I call the Jobs API to retrieve the deployed job, the finance task does not include those attributes:
disabled : false
email_notifications {0}
run_if : ALL_SUCCESS
run_job_task {2}
job_id : 513004093798668+018226-06-15T09:09:58.668Z
job_parameters {4}
ingestion_date : {{job.parameters.ingestion_date}}
reference_period_yyyymm : {{job.parameters.reference_period_yyyymm}}
reprocess_reason : {{job.parameters.reprocess_reason}}
table_suffix : {{job.parameters.table_suffix}}
task_key : finance
timeout_seconds : 0
I also checked the task in the Databricks UI, but I cannot see an option to configure Retries for a Run Job task. That option does appear for notebook tasks. The job is managed by a bundle and has edit_mode: UI_LOCKED. The parent job uses a scheduled trigger, not a continuous trigger.
17 hours ago
Hi alejandrorr
Thanks for checking all of that, it rules out the bundle. bundle validate shows the retry fields at the task level, the deployed bundle files have them, and the Jobs API returns the task without them. So the backend is silently dropping retry settings on this Run Job task. The missing Retries option in the UI points the same way.
This doesnโt match the Run Job task documentation, which says retries can be configured under Advanced task settings. Iโd raise a support ticket with the job ID, the bundle validate output and the API get response. Either the feature isnโt enabled in your workspace yet, or the docs are ahead of the platform.
Two quick checks before you open the ticket:
In the meantime, the most reliable workaround is to put the retries on the tasks inside the child job (p06_pipeline_monthly_finance) rather than on the Run Job task. Thatโs often the better design anyway, because only the failing task reruns instead of the whole child pipeline:
resources:
jobs:
p06_pipeline_monthly_finance:
tasks:
- task_key: <task_that_fails>
max_retries: 10
min_retry_interval_millis: 43200000
retry_on_timeout: true
# existing task settingsIf you need the retry at the parent level specifically, another option is to replace the Run Job task with a small notebook task that triggers the child job through the Databricks SDK (run_now_and_wait), raises an exception if the child run doesnโt succeed, and carries the retry policy itself. Notebook tasks do support retries, so the parent retries the whole child run that way.