cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

What is the timeout for dbutils.notebook.run, timeout = 0 ?

berserkersap
Contributor

Hello everyone,

I have several notebooks (around 10) and I want to run them in a sequential order. At first I thought of using %run but I have a variable that is repeatedly used in every notebook.

So now I am thinking to pass that variable from one main notebook (so that it is easier to change that variable manually only at one place instead of changing that in every notebook variable is being used)

dbutils.notebook.run(path = "test2", arguments={"current_year": current_year }, timeout_seconds = 0)

However, I found in the documentation that this command will fail if the notebook takes more than 10 min irrespective of the timeout we declare.

Documentation Source

image

So, I want to know will the command work even when the notebook takes more than 10 min.

When I checked this command using a 13 min notebook, the dbutils.notebook.run worked?

Sometimes, my notebook might take more than an hour so need some suggestions.

3 REPLIES 3

UmaMahesh1
Honored Contributor III

Hi @pavan venkata​ 

Yes, as the document says 0 means no timeout. It means that the notebook will take it's sweet time to complete execution without throwing an error due to a time limit. Be it if the notebook takes 1 min or 1 hour or 1 day or more. However, there is a limit of 30 days for job runs. you can find that in the same documentation place.

That 10 minutes you are seeing is for the databricks downtime. It has nothing to do with your code because obviously you can't do anything when databricks service itself is not working.

Cheers..

Thank You very much for replying @Uma Maheswara Rao Desula​ .

So, in the end it means it is alright to use it as long as the notebook doesn't take too long.

Also, Can you explain a bit more about databricks downtime. Are you talking about when databricks does maintenance or is it about the cluster being terminated ?

Thank You

UmaMahesh1
Honored Contributor III

Yes...It no longer cares about how long the notebook runs. As for the downtime, it is when there is a downtime in databricks service. If cluster gets terminated, you would receive a error for cluster termination itself.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!