topic Databricks Workflow if else task type in Data Engineering

Databricks Workflow if else task type

suryateja405555 — Tue, 24 Sep 2024 12:34:10 GMT

Need help on databricks workflow if else task.

Task-A : Notebook : executing set of the pyspark commands and loading the data into delta table.

I need to setup a custom status so that i can use status in the next if/else task.

Task-b: if/else: once i get the status of the previous task based on the status i need execute the set of notebook on True condition else stop the workflow.

need help on how to set and get the status of the previous notebook to current if/else task.

@databricksworkflow

Re: Databricks Workflow if else task type

szymon_dybczak — Tue, 24 Sep 2024 12:53:49 GMT

Hi @suryateja405555 ,

You can use task values in Task A to setup a custom status and then use that status in the if/else task to decide if you need to execute the of notebook on True condition or stop the workflow.

https://docs.databricks.com/en/jobs/share-task-context.html#using-task-values

Re: Databricks Workflow if else task type

suryateja405555 — Tue, 24 Sep 2024 13:15:35 GMT

but what is if/else expression we need to write to get the values from the previous task,

i tried with "tasks.previous_task_name.values.value_of_previous_task" =="SUCCESS" in the expression. throwing an error task value referenced by tasks.<previous_task>.values.value was not found.

i tried to declare/set a custom status by using these approaches.

1) dbutils.notebook.exit({status:"success"}) 2) dbutils.job.taskValues.get(key="status",values ="success")

Need your help to write if/else expression to get the value.

Thanks !!!

Re: Databricks Workflow if else task type

szymon_dybczak — Tue, 24 Sep 2024 13:50:09 GMT

You have error in your code. To set a task value you should use dbutils.job.taskValues.set command. Instead, in code you've provided you're using dbutils.job.taskValues.get.
Look at below picture. Here I have sample notebook where I'm setting status to "success"

Then I created sample workflow:

In if/else task I'm reffering to task value in following way: