cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

For each: nested JAR task can’t reference upstream taskValues

javasquez
New Contributor

I’m creating a For each with a nested JAR task. 
init_job (notebook) sets dbutils.jobs.taskValues.set("my_value", ...), and the For each container depends on init_job.

but the UI says: “Reference … can only be used in a task that depends on task 'init_job'.” This seems wrong because the nested task has a transitive dependency via the For each.

javasquez_0-1755874934680.png

If I switch the JAR nested task to Notebook Task, it can read the value correctly. for example:

Is this a bug/known limitation for JAR tasks inside For each?

javasquez_1-1755874991786.png

1 REPLY 1

WiliamRosa
New Contributor II

Yep—this is a current Workflows limitation. Task value refs ({{tasks.<key>.values.<name>}}) only work if the consuming task directly depends on the producer. A For Each/group dependency doesn’t count—nested tasks don’t inherit those edges for interpolation. That’s why the UI complains even though the group depends on init_job.

Wiliam Rosa
Data Engineer | Machine Learning Engineer
LinkedIn: linkedin.com/in/wiliamrosa

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now