topic Re: How to import one databricks python notebook into another? in Data Engineering

How to import one databricks python notebook into another?

amanpreetkaur — Mon, 15 Apr 2019 10:29:38 GMT

I have a python notebook A in Azure Databricks having import statement as below:

import xyz, datetime,...

I have another notebook xyz being imported in notebook A as shown in above code. When I run notebook A, it throws the following error:

ImportError:No module named xyz

Both notebooks are in the same workspace directory. Can anyone help in resolving this?

Re: How to import one databricks python notebook into another?

amanpreetkaur — Mon, 15 Apr 2019 12:23:19 GMT

Can you please answer this? @jurmu

Re: How to import one databricks python notebook into another?

jbsilva — Wed, 23 Oct 2019 16:29:18 GMT

I belive this is not possible. You can run a notebook passing parameters as:

result = dbutils.notebook.run("notebook-name", 60, {"argument": "data", "argument2": "data2", ...})

But notebooks can only return strings. What you need is to upload a Python module as a library and them import the module in your notebook.

Re: How to import one databricks python notebook into another?

MikhailKolomaso — Mon, 23 Dec 2019 11:18:14 GMT

For me worked well solution:

1) Create library notebook. For example - "Lib" with any functions/classes there (no runnable code).

2) Create main notebook. For example - "Main"

3) To import into main all classes & functions from Lib to Main use command:

%run "./Lib"

(this will works like: from Lib import *)

4) After that you can call any functions/ use classes that used in Lib from Main notebook.

PS:

1.1) recursion enabled - i.e. you lib notebook may contain code that runs any other notebooks the same way

1.2) for reload changed code from Lib module - just re-run command %run "./Lib"

Re: How to import one databricks python notebook into another?

awaiskaleem — Mon, 30 Dec 2019 11:42:00 GMT

@Mikhail Kolomasov , I tried exactly this, it worked until step 4. I'm trying to import variables in a py file into my "Main" notebook. What am i missing here?

2019-12-30-12-18-45-emr-pseg.png

Re: How to import one databricks python notebook into another?

m96 — Mon, 14 Sep 2020 15:36:24 GMT

Created a package to do exactly this. Have a look: https://pypi.org/project/libify/

(Relies on dbutils.notebook.run, so won't work with the Databricks community edition)

Re: How to import one databricks python notebook into another?

virgilus — Thu, 22 Apr 2021 15:30:34 GMT

Thank you it worked like a charm! 🙂

Re: How to import one databricks python notebook into another?

virgilus — Thu, 22 Apr 2021 15:31:12 GMT

Thanks, worked like a charm! 🙂

Re: How to import one databricks python notebook into another?

ramravi — Sun, 25 Dec 2022 15:18:19 GMT

NoebookA:

def func1():
 
     pass

NotebookB:

%run /NotebookA
 
func1()

Re: How to import one databricks python notebook into another?

artsheiko — Thu, 11 May 2023 14:46:41 GMT

Create a repository containing an __init__.py file
Add your library as .py file(s). Let's imagine that our library is composed by multiple sub-folders consolidated in "my_folder", one of sub-folders is named as "math_library" and contains my_awesome_lib.py
In caller Notebook import as a module :

import os
import sys
sys.path.append(os.path.abspath("/Workspace/Repos/<usename>/<repo-name>/<path>/my_folder"))

or use pathlib to adress the folder you need

import os
import sys
from pathlib import Path
 
sys.path.append(str(Path(os.getcwd()).parent.absolute()))

4. Make a call to the function in question :

from math_library import my_awesome_lib
 
my_awesome_lib.<function needed name>()

Re: How to import one databricks python notebook into another?

Mumrel — Fri, 12 May 2023 19:11:36 GMT

Why does the description here look much simpler than described above?

Re: How to import one databricks python notebook into another?

kinka007 — Sun, 14 May 2023 03:52:23 GMT

savastan0 cc shop

Re: How to import one databricks python notebook into another?

artsheiko — Sun, 14 May 2023 18:40:28 GMT

The documentation article covers the case when the file to import is located in the same directory. The step-by-step process I described shows how to import a file from an independent repository. Both are same in principle

Re: How to import one databricks python notebook into another?

Mumrel — Mon, 15 May 2023 07:30:31 GMT

Got it, thanks for the quick response

Re: How to import one databricks python notebook into another?

TimThein — Tue, 22 Aug 2023 22:01:24 GMT

I followed those descriptions and got the error "[Errno 95] Operation not supported" on the import line using DBR 12.2 LTS. I think there may be a bug in the runtime or the content of the descriptions is obsolete.