How can I trigger the execution of a specific step within a Databricks Workflow job?

[taschi]

I'm investigating methods to test a Job starting from a particular step. For instance, if I've made modifications midway through a 50+ step Job, is there a way to test the Job without running the steps that precede the one with the modification?

[nkraj]

Hello @taschi ,

The Repair Job Run feature can be valuable in scenarios where you need to fix failed or canceled multi-task jobs. It enables you to rerun only the subset of tasks that were unsuccessful along with any dependent tasks.

Please consult the provided reference article to explore this feature further: Repair Job Run

[taschi]

Hello @nkraj, appreciate your response. I've come across that feature, but my specific interest lies in a scenario where we have a new version of the job with no previous failure runs. In this case, we aim to test it, but rather than starting from the beginning, we want to initiate it from a specific step to optimize testing costs.

[nkraj · Accepted solution]

Hello @taschi ,

Appreciate the thorough use case explanation. Currently, rerunning a new version of a job with the option to select specific tasks is not feasible. However, its on the roadmap to enable users to choose any task for rerun, even those that were successful.

[taschi]

Got it, thanks for clarifying!

[SamAdams]

@nkraj is there a product board where we can vote for this feature to trigger a specific task outside of a repair run?

[alan-nousot]

Really interested in this feature. I'd love to be able to programmatically orchestrate tasks with more granularity.