Hi @Retired_mod for your quick reply. I will test it out in our scenario and let you know. Just for confirmation if I have two scripts (e.g. ingest.py and train.py) and in my task named "ingest" I do something like inside ingest.py I run:
dbutils.jobs.taskValues.set(taskKey = "ingest", key = "processed_data", value=data)

then should I pass inside the pipeline for the train.py: {{tasks.ingest.values.processed_data}}?