topic Delay in retrieving task variables from a notebook utilizing %run in Data Engineering

Delay in retrieving task variables from a notebook utilizing %run

cmilligan — Thu, 09 Feb 2023 15:32:35 GMT

I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call that notebook using %run.

In the tasks immediately following the the notebook where I am setting the variables and assigning them I'm getting failures in the %run command due to the variables not being set. I've tried using time.sleep() in the notebook that I'm setting the variables but that hasn't worked. I never had this issue when calling them directly in the notebook, just when calling them in a separate notebook using %run.

Is there a known issue where there is a delay between when a variable is set and how soon after it can be called.

Re: Delay in retrieving task variables from a notebook utilizing %run

Lakshay — Fri, 10 Feb 2023 19:29:54 GMT

Hi @Coleman Milligan , How are you setting the variables in the shared notebook? Could you please share some code to understand better?

Re: Delay in retrieving task variables from a notebook utilizing %run

Debayan — Wed, 15 Feb 2023 17:40:23 GMT

Hi, Please refer: https://docs.databricks.com/notebooks/notebook-workflows.html

Could you please provide the error received?