topic Re: How to disable task in databricks job workflow in Get Started Discussions

How to disable task in databricks job workflow

shubbansal27 — Sun, 16 Jul 2023 06:19:13 GMT

I don't see any option to disable the task. I have created one workflow which contains multiple tasks.

sometime i need to disable some task for my experiment. but there is no straightforward way for this. I feel "disable" button is must have option for task settings.

currently i am using one workaround but this is not the good way. please suggest the fix

whenever i need to disable the task. i just add "dbutils.notebook.exit" function call in the top of task notebook to skip the further execution. but this doesn't prevents the cluster to spawn. this is wastage of resource and also adds delay in workflow.

Re: How to disable task in databricks job workflow

User16539034020 — Wed, 19 Jul 2023 17:02:34 GMT

Hi, shubbansal27:

Thanks for contacting Databricks Support!

At the moment, this can only be achieved using the if/else (conditional) task type. For this, you would need to establish a variable to denote whether you want a task to be paused and the task type to implement the verification before running the task. The promising news is that our engineering team is actively exploring ways to simplify this process. You can anticipate a more user-friendly feature to be introduced in the near future.

Monica

Re: How to disable task in databricks job workflow

User16539034020 — Wed, 19 Jul 2023 17:14:28 GMT

Hi, shubbansal27:
Thanks for contacting Databricks Support!

Re: How to disable task in databricks job workflow

Siebert_Looije — Sat, 22 Jul 2023 09:21:21 GMT

Hi,
Thanks for your question. Right now there is no way of doing this within the workflow.

So the way we are doing this now, is passing through a task value in the workflow and checking the task value at the beginning of the notebook and use the dbutils.notebook.exit to

Re: How to disable task in databricks job workflow

Anonymous — Sun, 23 Jul 2023 04:05:43 GMT

Hi @shubbansal27

We haven't heard from you since the last response from @Siebert_Looije , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Re: How to disable task in databricks job workflow

shubbansal27 — Sun, 23 Jul 2023 05:57:18 GMT

Hi Siebert,

Thanks for reply on my post. your understanding is correct. this way is skipping the task to run further.. but this doesn't stop the cluster to spawn which i have assigned for the task.

Actually i want to attach different run schedule to individual task but currently schedule is at job level.

ex: in my workflow, there are ML training tasks and prediction tasks. i dont want to run training tasks daily.

Re: How to disable task in databricks job workflow

shubbansal27 — Sun, 23 Jul 2023 05:59:16 GMT

Hi Vidula,

I am looking for solution for schedule option at task level. or task disable option.

Re: How to disable task in databricks job workflow

sruthisatish3 — Wed, 07 Feb 2024 11:23:15 GMT

Hi,

I am also facing a similar issue where I want to disable tasks according to the environment where the job runs.For ex : if the environment is dev ,run only Task 1 else run Task 2 and Task3.

Could you please help me understand how that variable can be used to pause the task ?Currently, I am not able to find any options that will disable the task.

Re: How to disable task in databricks job workflow

ksievers — Thu, 30 May 2024 15:31:35 GMT

Can you let us know when this " more user-friendly feature" to disable a specific task in a workflow will be available?

Re: How to disable task in databricks job workflow

VickiT — Tue, 08 Oct 2024 07:54:20 GMT

I know this post was a while ago, but I've hit this issue, and come up with using if/else conditions between the tasks with a Job parameter of "IsActiveTask2" with a True or False, and then I can have the Task2 dependant on the success of this conditional step being True. I hope that makes sense?