cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to import one databricks python notebook into another?

amanpreetkaur
New Contributor III

I have a python notebook A in Azure Databricks having import statement as below:

import xyz, datetime,...

I have another notebook xyz being imported in notebook A as shown in above code. When I run notebook A, it throws the following error:

ImportError:No module named xyz  

Both notebooks are in the same workspace directory. Can anyone help in resolving this?

14 REPLIES 14

amanpreetkaur
New Contributor III

Can you please answer this? @jurmu

jbsilva
New Contributor II

I belive this is not possible. You can run a notebook passing parameters as:

result = dbutils.notebook.run("notebook-name", 60, {"argument": "data", "argument2": "data2", ...})

But notebooks can only return strings. What you need is to upload a Python module as a library and them import the module in your notebook.

MikhailKolomaso
New Contributor II

For me worked well solution:

1) Create library notebook. For example - "Lib" with any functions/classes there (no runnable code).

2) Create main notebook. For example - "Main"

3) To import into main all classes & functions from Lib to Main use command:

%run "./Lib"

(this will works like: from Lib import *)

4) After that you can call any functions/ use classes that used in Lib from Main notebook.

PS:

1.1) recursion enabled - i.e. you lib notebook may contain code that runs any other notebooks the same way

1.2) for reload changed code from Lib module - just re-run command %run "./Lib"

@Mikhail Kolomasov​ , I tried exactly this, it worked until step 4. I'm trying to import variables in a py file into my "Main" notebook. What am i missing here?

2019-12-30-12-18-45-emr-pseg.png

Thank you it worked like a charm! 🙂

Thanks, worked like a charm! 🙂

m96
New Contributor II

Created a package to do exactly this. Have a look: https://pypi.org/project/libify/

(Relies on dbutils.notebook.run, so won't work with the Databricks community edition)

ramravi
Contributor II

NoebookA:

def func1():
 
     pass

NotebookB:

%run /NotebookA
 
func1()

artsheiko
Databricks Employee
Databricks Employee
  1. Create a repository containing an __init__.py file
  2. Add your library as .py file(s). Let's imagine that our library is composed by multiple sub-folders consolidated in "my_folder", one of sub-folders is named as "math_library" and contains my_awesome_lib.py
  3. In caller Notebook import as a module :
import os
import sys
sys.path.append(os.path.abspath("/Workspace/Repos/<usename>/<repo-name>/<path>/my_folder"))

or use pathlib to adress the folder you need

import os
import sys
from pathlib import Path
 
sys.path.append(str(Path(os.getcwd()).parent.absolute()))

4. Make a call to the function in question :

from math_library import my_awesome_lib
 
my_awesome_lib.<function needed name>()

Why does the description here look much simpler than described above?

artsheiko
Databricks Employee
Databricks Employee

The documentation article covers the case when the file to import is located in the same directory. The step-by-step process I described shows how to import a file from an independent repository. Both are same in principle

Got it, thanks for the quick response

Tim3
New Contributor II

I followed those descriptions and got the error "[Errno 95] Operation not supported" on the import line using DBR 12.2 LTS. I think there may be a bug in the runtime or the content of the descriptions is obsolete.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group