topic Re: How can I trigger the execution of a specific step within a Databricks Workflow job? in Data Engineering

How can I trigger the execution of a specific step within a Databricks Workflow job?

taschi — Fri, 26 Jan 2024 12:53:42 GMT

I'm investigating methods to test a Job starting from a particular step. For instance, if I've made modifications midway through a 50+ step Job, is there a way to test the Job without running the steps that precede the one with the modification?

Re: How can I trigger the execution of a specific step within a Databricks Workflow job?

nkraj — Fri, 26 Jan 2024 13:42:51 GMT

Hello @taschi ,

The Repair Job Run feature can be valuable in scenarios where you need to fix failed or canceled multi-task jobs. It enables you to rerun only the subset of tasks that were unsuccessful along with any dependent tasks.

Please consult the provided reference article to explore this feature further: Repair Job Run

Re: How can I trigger the execution of a specific step within a Databricks Workflow job?

taschi — Fri, 26 Jan 2024 14:30:35 GMT

Hello @nkraj, appreciate your response. I've come across that feature, but my specific interest lies in a scenario where we have a new version of the job with no previous failure runs. In this case, we aim to test it, but rather than starting from the beginning, we want to initiate it from a specific step to optimize testing costs.

Re: How can I trigger the execution of a specific step within a Databricks Workflow job?

nkraj — Fri, 26 Jan 2024 15:25:04 GMT

Hello @taschi ,

Appreciate the thorough use case explanation. Currently, rerunning a new version of a job with the option to select specific tasks is not feasible. However, its on the roadmap to enable users to choose any task for rerun, even those that were successful.

Re: How can I trigger the execution of a specific step within a Databricks Workflow job?

taschi — Mon, 29 Jan 2024 11:41:27 GMT

Got it, thanks for clarifying!

Re: How can I trigger the execution of a specific step within a Databricks Workflow job?

SamAdams — Tue, 17 Sep 2024 16:10:32 GMT

@nkraj is there a product board where we can vote for this feature to trigger a specific task outside of a repair run?

Re: How can I trigger the execution of a specific step within a Databricks Workflow job?

alan-nousot — Wed, 11 Dec 2024 23:20:55 GMT

Really interested in this feature. I'd love to be able to programmatically orchestrate tasks with more granularity.

Re: How can I trigger the execution of a specific step within a Databricks Workflow job?

SamAdams — Thu, 01 May 2025 14:58:25 GMT

It's now generally available