cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Orchestrate jobs using a parameter set in a notebook

george_ognyanov
New Contributor III

I am trying to orchestrate my Databricks Workflows tasks using a parameter I would set in a notebook.

Given the workflow below I am trying to set a parameter in the Cinderella task which is a python notebook. Once set I would like to use this parameter in the true_false_selection block instead of this 1=1 condition. Something like:

if ${notebook_parameter} == "Rolling" --> execution continues to the isTrue task which is also a python notebook

elif ${notebook_parameter} == "Prophet" --> execution continues to the isNotTrue task which is another python notebook. 

george_ognyanov_0-1701258224857.png

Precisely, I want to use parameter set inside notebook in this menu selection as the pseudo code shows: 

george_ognyanov_0-1701259276034.png

I am aware of both 
dbutils.jobs.taskValues.set(key='selection', value='Rolling')
&& 
dbutils.widgets.text("selection", "Rolling")
but it appears neither can be used in the if/else condition task. 

 

1 REPLY 1

Panda
New Contributor II

Here's how we can proceed, follow the instructions below:

In your previous task, depending on whether you're using Python or Scala, set the task value like this:
dbutils.jobs.taskValues.set("check_value", "2")

In your if-else task, you must reference the name of the previous task as follows:
{{tasks.['previous_task'].values.['check_value']}}

Please review the attached screenshot for further clarification

Panda_0-1710635500605.png

 



Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!