cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How can I trigger the execution of a specific step within a Databricks Workflow job?

taschi
New Contributor III

I'm investigating methods to test a Job starting from a particular step. For instance, if I've made modifications midway through a 50+ step Job, is there a way to test the Job without running the steps that precede the one with the modification?

1 ACCEPTED SOLUTION

Accepted Solutions

nkraj
Databricks Employee
Databricks Employee

Hello @taschi ,

Appreciate the thorough use case explanation. Currently, rerunning a new version of a job with the option to select specific tasks is not feasible. However, its on the roadmap to enable users to choose any task for rerun, even those that were successful.

View solution in original post

6 REPLIES 6

nkraj
Databricks Employee
Databricks Employee

Hello @taschi ,

The Repair Job Run feature can be valuable in scenarios where you need to fix failed or canceled multi-task jobs. It enables you to rerun only the subset of tasks that were unsuccessful along with any dependent tasks.

Please consult the provided reference article to explore this feature further: Repair Job Run

taschi
New Contributor III

Hello @nkraj, appreciate your response. I've come across that feature, but my specific interest lies in a scenario where we have a new version of the job with no previous failure runs. In this case, we aim to test it, but rather than starting from the beginning, we want to initiate it from a specific step to optimize testing costs.

nkraj
Databricks Employee
Databricks Employee

Hello @taschi ,

Appreciate the thorough use case explanation. Currently, rerunning a new version of a job with the option to select specific tasks is not feasible. However, its on the roadmap to enable users to choose any task for rerun, even those that were successful.

taschi
New Contributor III

Got it, thanks for clarifying!

SamAdams
New Contributor III

@nkraj is there a product board where we can vote for this feature to trigger a specific task outside of a repair run?

alan-nousot
Visitor

Really interested in this feature. I'd love to be able to programmatically orchestrate tasks with more granularity.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group