topic Re: API for Restarting Individual Failed Tasks within a Job? in Data Engineering

API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Wed, 16 Apr 2025 04:13:27 GMT

Hi everyone,

I'm exploring ways to streamline my workflow in Databricks and could really use some expert advice. In my current setup, I have a job (named job_silver) with multiple tasks (e.g., task 1, task 2, task 3). When one of these tasks fails—say task 2—I want the ability to restart just that specific task without rerunning the entire job.

I did some research and came across the “Repair and Rerun” feature (Databricks Blog). While that's a great tool for saving time and money in data and ML workflows, my use case requires more flexibility. Specifically, I'm looking for an API-based solution that I can integrate into my code, allowing dynamic control over which task to restart based on custom logic.

Some points I’m particularly interested in:

Is there an existing API (or a combination of APIs) that allows for restarting individual tasks within a job?
Could this be done via the REST API, and if so, what endpoints or methods should I look at?
Are there any workarounds or best practices for implementing this functionality if a dedicated API is not available?
How might this approach scale in environments with a large number of jobs and complex dependency graphs?

I’d love to hear about your experiences and any code snippets or documentation pointers that could help me get started. Thanks in advance for your insights!

Re: API for Restarting Individual Failed Tasks within a Job?

Aviral-Bhardwaj — Wed, 16 Apr 2025 04:27:26 GMT

use retry job or task option it is available bottom right side of the job or task plane

Re: API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Wed, 16 Apr 2025 04:39:40 GMT

Hi @Aviral-Bhardwaj ,

Thanks for your suggestion. But I really need is a code/API-driven approach rather than having to click around in the UI to manage a specific task in a job retry.

Re: API for Restarting Individual Failed Tasks within a Job?

Aviral-Bhardwaj — Wed, 16 Apr 2025 04:41:41 GMT

https://docs.databricks.com/api/workspace/jobs/reset

Re: API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Wed, 16 Apr 2025 04:53:28 GMT

Thank you so much for sharing that information. After reviewing the /api/2.2/jobs/reset API documentation, it looks like this endpoint is designed for updating the entire job configuration for future runs and doesn’t support restarting only the failed tasks of an active job run.

I appreciate your help, and if you have any other suggestions or ideas on achieving that level of granularity, I'd be happy to hear them.

Re: API for Restarting Individual Failed Tasks within a Job?

RiyazAliM — Wed, 16 Apr 2025 08:20:58 GMT

hi @minhhung0507

Please check if the below code using rest api is able to cater to your needs.

list_runs_url = f"{DATABRICKS_INSTANCE}/api/2.1/jobs/runs/list"

params = {

"job_id": JOB_ID,

"limit": 1,

"active_only": False

}

response = requests.get(list_runs_url, headers=headers, params=params)

latest_run_id = response.json()["runs"][0]["run_id"]

run_details_url = f"{DATABRICKS_INSTANCE}/api/2.1/jobs/runs/get"

response = requests.get(run_details_url, headers=headers, params={"run_id": latest_run_id})

tasks = response.json().get("tasks", [])

failed_tasks = [task["task_key"] for task in tasks if task["state"]["result_state"] == "FAILED"]

print(failed_tasks)

if not failed_tasks:

print("No failed tasks found.")

else:

print(f"Retrying failed tasks: {failed_tasks}")

# Submit new run for each failed task (assuming same notebook and cluster setup)

for task in tasks:

if task["task_key"] in failed_tasks:

submit_url = f"{DATABRICKS_INSTANCE}/api/2.1/jobs/runs/submit"

payload = {

"run_name": f"Rerun failed task: {task['task_key']}",

"tasks": [

{

"task_key": task["task_key"],

"notebook_task": task["notebook_task"],

"existing_cluster_id": task["existing_cluster_id"]

}

]

}

submit_resp = requests.post(submit_url, headers=headers, data=json.dumps(payload))

print(f"Submitted rerun for task {task['task_key']}: {submit_resp.status_code} - {submit_resp.text}")

Re: API for Restarting Individual Failed Tasks within a Job?

RiyazAliM — Wed, 16 Apr 2025 14:40:26 GMT

Hey @minhhung0507

Keep me posted on how this works for you.

Looking forward

Cheers!

Re: API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Thu, 17 Apr 2025 02:44:50 GMT

Sure, we will try again based on the solution you provided and will feedback to you whether it works or not. Thank you very much.

Re: API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Thu, 17 Apr 2025 03:45:05 GMT

Hi @RiyazAliM , after applying your code to the pipeline it seems that it can't work because of the following error:

400 --- {"error_code":"INVALID_PARAMETER_VALUE","message":"One of job_cluster_key, new_cluster, or existing_cluster_id must be specified. Serverless compute for workflows is not enabled in the workspace.","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"74ef6feb-9cff-4d3d-8cb6-3f45f3525768","serving_data":""}]}

Could you plese help us investigate this error?

Re: API for Restarting Individual Failed Tasks within a Job?

RiyazAliM — Thu, 17 Apr 2025 05:43:07 GMT

Hey @minhhung0507 - quick question - what is the cluster type you're using to run your workflow?

I'm using a shared, interactive cluster, so I'm passing the parameter

{'existing_cluster_id' : task['existing_cluster_id']}

in the payload. This parameter will change if you're using a job_cluster, I'd have to check what it's gonna be for serveless though.

Hope it helps.

Cheers!

Re: API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Thu, 17 Apr 2025 09:25:55 GMT

Hi @RiyazAliM ,

Thanks for showing how to pass the existing_cluster_id in the payload. One thing I’m worried about from a production standpoint is what happens if the ID we supply doesn’t correspond to an active cluster—will Databricks automatically spin up a new cluster under the hood, or will the run fail immediately?

Auto‑provisioning clusters without explicit control could lead to unwanted resource sprawl and cost overruns. Could you please clarify the behavior of the Submit Run API when existing_cluster_id can’t be resolved, and recommend best practices to ensure we only target known, managed clusters?

Re: API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Thu, 17 Apr 2025 11:09:44 GMT

@RiyazAliM FYI

@RiyazAliM FYI:

400 --- {"error_code":"INVALID_PARAMETER_VALUE","message":"The cluster 0417-102632-cuhsyfm0 is not an all-purpose cluster. existing_cluster_id only supports all-purpose cluster IDs.","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"3379c0ea-0fc0-405a-808c-7f06759aad83","serving_data":""}]}

Re: API for Restarting Individual Failed Tasks within a Job?

RiyazAliM — Mon, 21 Apr 2025 08:07:36 GMT

Hi @minhhung0507

The trigger won't be successful if the cluster-id is not provided. I tried to replicate it and found that it throws an error as below:

{"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required fields: settings.job_clusters.job_cluster_key, settings.job_clusters.new_cluster","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"b9631652-3101-4b69-95a6-45be477c85a5","serving_data":""}]}

Re: API for Restarting Individual Failed Tasks within a Job?

RiyazAliM — Mon, 21 Apr 2025 08:10:53 GMT

The error depicts that you've passed job_cluster id with the key as existing_cluster_id which is used for interactive clusters. Please pass job_cluster_details if you're using the key `

"job_cluster_key"`. Thank you

Re: API for Restarting Individual Failed Tasks within a Job?

minhhung0507 — Mon, 21 Apr 2025 08:50:27 GMT

Hi @RiyazAliM ,

I have two quick questions I’d appreciate your help with:

Why do we need to use an interactive (all‑purpose) cluster when submitting a job, rather than a job cluster?
I can’t find any reference to a parameter called job_cluster_details in the official docs—could you point me to where it’s documented or share a link?

Thanks in advance for any pointers!

Re: API for Restarting Individual Failed Tasks within a Job?

RiyazAliM — Thu, 24 Apr 2025 11:19:45 GMT

Hey @minhhung0507
Great questions, let me answer as per my understanding:

Q1: Why do we need to use an interactive (all‑purpose) cluster when submitting a job, rather than a job cluster?

As I'm testing, it's a matter of convenience, nothing more, I couldn't wait to run the job till a job cluster spins up, hence using an interactive cluster. As I figured out you're using a job cluster to pass the request, I suggested to use a different parameter in the JSON payload.

Q2: I can’t find any reference to a parameter called job_cluster_details in the official docs—could you point me to where it’s documented or share a link?

I was checking the RestAPI docs and from the link below, saw how the job_cluster is being referenced. Suggested the same to you.

In the example above, "auto_scaling_cluster" is the name of the job cluster.

Link to the doc - https://docs.databricks.com/api/workspace/jobs/get

Also, check this API for repair run, I believe this would be the right URL to hit for your usecase.
https://docs.databricks.com/api/workspace/jobs/repairrun

Let me know your thoughts.