topic Re: Is there a way to run notebooks concurrently in same session? in Data Engineering

Is there a way to run notebooks concurrently in same session?

darshan — Mon, 27 Jun 2022 13:29:42 GMT

tried using-

dbutils.notebook.run(notebook.path, notebook.timeout, notebook.parameters)

but it takes 20 seconds to start new session. %run uses same session but cannot figure out how to use it to run notebooks concurrently.

Re: Is there a way to run notebooks concurrently in same session?

Hubert-Dudek — Mon, 27 Jun 2022 14:10:17 GMT

dbutils.notebook.run starts a new job, that's why it takes this time and test yo can start multiple concurrently using ThreadPool or other async libraries. Probably with the better server, it could be 10 seconds, not 20.

%RUN executes other notebooks like it would be code of notebook which we run (like include in some languages).

Maybe you could plan a different architecture way (but it depends on the use case) as this way is instead for no-time critical projects (analytics). Stream and delta live tables should be used instead if it is a project when that 20s delay is too much.

Re: Is there a way to run notebooks concurrently in same session?

craig_lukasik — Mon, 27 Jun 2022 19:47:20 GMT

Another approach is to use the Jobs API and leverage a notebook_task with an existing cluster (existing_cluster_id).

Re: Is there a way to run notebooks concurrently in same session?

Prabakar — Tue, 28 Jun 2022 08:22:00 GMT

You can use jobs workflow. It helps you to run the notebooks in parallel as well as sequential.

Re: Is there a way to run notebooks concurrently in same session?

bb_huey99 — Tue, 28 Jun 2022 17:53:42 GMT

This is a great question. I’ve been struggling with opening multiple browser sessions to open more than one notebook at a time.

Re: Is there a way to run notebooks concurrently in same session?

sbo — Tue, 28 Jun 2022 18:33:19 GMT

dbutils.notebook.run will run multiple notebooks in the same session!

Re: Is there a way to run notebooks concurrently in same session?

darshan — Wed, 29 Jun 2022 07:22:02 GMT

Hello, yes responses were helpful, so I tried workflow but stuck with sending data across notebooks using workflow.

Also, trying delta live tables. What I would like to know is is there threshold about how many records delta live tables can hold at a time?

Re: Is there a way to run notebooks concurrently in same session?

darshan — Wed, 29 Jun 2022 07:23:26 GMT

But it creates new jobs which makes process slow by 20 seconds

Re: Is there a way to run notebooks concurrently in same session?

darshan — Wed, 29 Jun 2022 07:24:26 GMT

Is there a way to send data from one notebook to other using workflow?

Re: Is there a way to run notebooks concurrently in same session?

darshan — Wed, 29 Jun 2022 11:28:42 GMT

Hello, one thing I am not able to understand is, while developing low latency application where even 5 seconds cannot be tolerated, how to create jobs? Because each job takes 20 seconds to init.

Re: Is there a way to run notebooks concurrently in same session?

zulazuardi — Wed, 29 Jun 2022 22:20:08 GMT

You can use python concurrent run and execute the dbutils.notebooks.run

Re: Is there a way to run notebooks concurrently in same session?

gfree76 — Wed, 29 Jun 2022 22:37:29 GMT

There is a databricks sample notebook that shows how to use scala futures to run notebooks in parallel. We use this extensively, so I can promise it works 🙂 Look for the section Run Notebooks Concurrently https://docs.databricks.com/notebooks/notebook-workflows.html

Re: Is there a way to run notebooks concurrently in same session?

darshan — Thu, 30 Jun 2022 07:01:31 GMT

This does help, but can you help me understand why job run time takes more time than actual run time of notebook? And is there a way to reduce the time?

Re: Is there a way to run notebooks concurrently in same session?

rudesingh56 — Mon, 04 Jul 2022 13:49:31 GMT

I’ve been struggling with opening multiple browser sessions to open more than one notebook at a time.