I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call that notebook using %run.
In the tasks immediately following the the notebook where I am setting the variables and assigning them I'm getting failures in the %run command due to the variables not being set. I've tried using time.sleep() in the notebook that I'm setting the variables but that hasn't worked. I never had this issue when calling them directly in the notebook, just when calling them in a separate notebook using %run.
Is there a known issue where there is a delay between when a variable is set and how soon after it can be called.
