Orchestrate jobs using a parameter set in a notebook

george_ognyanov — Wed, 29 Nov 2023 12:02:12 GMT

I am trying to orchestrate my Databricks Workflows tasks using a parameter I would set in a notebook.

Given the workflow below I am trying to set a parameter in the Cinderella task which is a python notebook. Once set I would like to use this parameter in the true_false_selection block instead of this 1=1 condition. Something like:

if ${notebook_parameter} == "Rolling" --> execution continues to the isTrue task which is also a python notebook

elif ${notebook_parameter} == "Prophet" --> execution continues to the isNotTrue task which is another python notebook.

Precisely, I want to use parameter set inside notebook in this menu selection as the pseudo code shows:

I am aware of both
dbutils.jobs.taskValues.set(key='selection', value='Rolling')
&&
dbutils.widgets.text("selection", "Rolling")
but it appears neither can be used in the if/else condition task.

Re: Orchestrate jobs using a parameter set in a notebook

Panda — Sun, 17 Mar 2024 00:34:38 GMT

Here's how we can proceed, follow the instructions below:

In your previous task, depending on whether you're using Python or Scala, set the task value like this:
dbutils.jobs.taskValues.set("check_value", "2")

In your if-else task, you must reference the name of the previous task as follows:
{{tasks.['previous_task'].values.['check_value']}}

Please review the attached screenshot for further clarification

topic Orchestrate jobs using a parameter set in a notebook in Data Engineering

Orchestrate jobs using a parameter set in a notebook

Re: Orchestrate jobs using a parameter set in a notebook