cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delay in retrieving task variables from a notebook utilizing %run

cmilligan
Contributor II

I have a databricks workflow where the first task is to set up task parameters for other notebooks to use later in the process. Since these are variables that are used in all of my notebooks, I have opted to assign them in a shared notebook and call that notebook using %run.

In the tasks immediately following the the notebook where I am setting the variables and assigning them I'm getting failures in the %run command due to the variables not being set. I've tried using time.sleep() in the notebook that I'm setting the variables but that hasn't worked. I never had this issue when calling them directly in the notebook, just when calling them in a separate notebook using %run.

Is there a known issue where there is a delay between when a variable is set and how soon after it can be called.

2 REPLIES 2

Lakshay
Esteemed Contributor
Esteemed Contributor

Hi @Coleman Milligan​ , How are you setting the variables in the shared notebook? Could you please share some code to understand better?

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi, Please refer: https://docs.databricks.com/notebooks/notebook-workflows.html

Could you please provide the error received?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!