topic Re: Long time turning on another notebook in Data Engineering

Long time turning on another notebook

LukaszJ — Fri, 21 Mar 2025 13:17:32 GMT

Hello,

I want to run some notebooks from notebook "A".

And regardless of the contents of the some notebook, it is run for a long time (20 seconds). It is constans value and I do not know why it takes so long.

I tried run simple notebook with one input parameter and only print it - it takes the same 20 seconds.

I use this method:

notebook_result = dbutils.notebook.run("notebook_name", 60, {"key1": "value1", "key2": "value2"})

The notebooks are in the same folder and in the same cluster (really good cluster).

Could someone explain me why it takes so long and how can I speed it run?

Best regards,

Łukasz

Re: Long time turning on another notebook

MartinB — Mon, 28 Feb 2022 16:59:35 GMT

I guess the creation of the spark session requires the 20 seconds

Re: Long time turning on another notebook

Ryan_Chynoweth — Mon, 28 Feb 2022 17:50:46 GMT

I believe that dbutils.notebook.run creates a new session so there is a little more overhead. If you do not want to create a new session you can use

%run <NOTEBOOK PATH>

This will execute the notebook inline with the same session as the parent notebook. Note that this shares the session so if you define variables or functions in the child notebook they will be available in the parent notebook.

Also, if you are trying to orchestrate notebooks you should use the task orchestration available in the Databricks jobs ui.

Re: Long time turning on another notebook

Hubert-Dudek — Tue, 01 Mar 2022 08:33:31 GMT

You can also just use files in repos and import needed library/class to your notebook.

If you run 2 notebooks in parallel it is good to reserve resources for every of them using pool option:

spark.sparkContext.setLocalProperty("spark.scheduler.pool", "notebook1")

Re: Long time turning on another notebook

LukaszJ — Tue, 01 Mar 2022 08:41:45 GMT

Hello Ryan,

Thank you for the response.

Now I understand.

However, is there any way to put inputs and take outputs from the notebook using this method?

Best regards,

Łukasz

Re: Long time turning on another notebook

LukaszJ — Fri, 04 Mar 2022 13:48:43 GMT

Hello Hubert,

Thank you for the response.

I am not sure if it works for me.

I run in a loop the same notebook in a few times. Something like that:

spark.sparkContext.setLocalProperty("spark.scheduler.pool", "My_Notebook")
 
for row in data:
    notebook_results = dbutils.notebook.run("My_Notebook", 60, {"data": row})

And yet the time to start any notebook is several seconds.

Could you tell me what is wrong with this solution?

Best regards,

Łukasz

Re: Long time turning on another notebook

Ryan_Chynoweth — Fri, 04 Mar 2022 17:01:58 GMT

I do not believe you can get outputs from dbutils.notebook.exit. But you could potentially drop a file locally with values and read it in the other notebook or save them as variables and access that variable.

Re: Long time turning on another notebook

LukaszJ — Wed, 09 Mar 2022 09:10:08 GMT

Okay I am not able to set the same session for the both notebooks (parent and children).

So my result is to use %run ./notebook_name .

I put all the code to functions and now I can use them.

Example:

# Children notebook
def do_something(param1, param2):
    # some code ...
    return result_value

# Parent notebook
 
# some code ...
 
%run ./children_notebook
 
# some code ...
 
function_result = do_something(value_1, value_2)
 
# some code ...

Thanks to everyone for the answers