Quick update, my question effectively boils down to:

Do databricks workflows have "global" variables that can be set programmatically from anywhere in the workflow (e.g. nested notebook task inside a parent run_job task) during runtime and be referenced anywhere else in the workflow, regardless of scope?

Consulting with LLMs, I have some partial answers but still would appreciate some feedback from the community!

Updates on my considered approaches:

• The first option I think wouldn't work as I was hoping due to variable scoping
• The second option seems like it's still a viable option, but the same challenges/trickiness persist
• Other options I've seen proposed elsewhere:
  • DBFS/Cloud Storage (e.g. file with runtime information saved and referenced elsewhere during job run)
  • External DB/Table (e.g. tasks read/write key-value pairs to a shared Delta table or external database)