topic How to import local python file in notebook? in Data Engineering

How to import local python file in notebook?

jsaddam28 — Fri, 04 Sep 2015 07:18:32 GMT

for example I have one.py and two.py in databricks and I want to use one of the module from one.py in two.py. Usually I do this in my local machine by import statement like below

two.py__

from one import module1

How to do this in databricks???

Re: How to import local python file in notebook?

vida — Fri, 04 Sep 2015 16:24:21 GMT

Just do a %run command of the other notebook which will import all the functions there.

%run your_folder/run2.py

-Vida

Re: How to import local python file in notebook?

JavierOrozco — Fri, 11 Sep 2015 16:23:58 GMT

Hi Vida,

I have the same problem, and I followed your solution but I get the error that the file cannot be found. I tried with and without full path when the run2.py is in the same folder of the notebook

Stacktrace: /Users/username/Test NB: python

Thanks for your assistance

Re: How to import local python file in notebook?

vida — Fri, 11 Sep 2015 17:27:51 GMT

Hi,

You need the full path - we don't support relative paths yet. I can do the following:

%run "/Users/XXXXXXX@databricks.com/InnerNotebook"

Please make sure you copy the notebook name correctly without any spaces, etc.

-Vida

Re: How to import local python file in notebook?

JavierOrozco — Fri, 11 Sep 2015 19:08:16 GMT

This the exact copy of the code

%run '/Users/admin/s3_handling_poc.py'
Notebook not found: /Users/admin/s3_handling_poc.py

Whereas with a Notebook it does work:

%run /Users/admin/test2
Command took 0.18s
Hello World

Re: How to import local python file in notebook?

vida — Fri, 11 Sep 2015 19:09:35 GMT

Can you try without the ".py" at the end?

What do you mean by with a notebook it does work?

Re: How to import local python file in notebook?

JavierOrozco — Fri, 11 Sep 2015 19:17:04 GMT

I doesn't seem to work either, I created a "Hello World" inside a python script. If I run it with ".py" shows same error, but without ".py" looks executing but never ends until I cancel it.

Re: How to import local python file in notebook?

vida — Fri, 11 Sep 2015 19:20:56 GMT

Sorry - I'm confused - is your file - s3_handling_poc.py uploaded to Databricks?

%run is for running one notebook within another Databricks notebook.

To get local Python code into Databricks - you'll need to either import your python file as a Databricks Notebook. Or you can create an egg from your python code and upload that as a library. If it's a single python file - importing it as a Databricks notebook is going to be the easier route.

Re: How to import local python file in notebook?

JavierOrozco — Fri, 11 Sep 2015 20:36:39 GMT

yes it is python code, I let it run and it managed to finish.

I was following the initial instruction of how to import third party python modules. The instruction was to run them. However the same code runs faster within a notebook. The problem is how do I use my own existing python code into new databricks python notebooks. Do I have to embed all of them into a notebook?

%run "/Users/admin/test_hello"

Command took 746.62s

HELLO WORLD FROM PYTHON

Re: How to import local python file in notebook?

PaulAgnew — Wed, 02 Mar 2016 19:37:48 GMT

@javier.orozco @realeuesit.com, did you ever get clarity or resolution to this problem, its as if dies with your last statement. The last two sentences of your comment sum the issue very nicely.

Re: How to import local python file in notebook?

dchokkadi1_5588 — Mon, 11 Apr 2016 21:46:59 GMT

@vida, how to run a notebook that is mounted with dbfs?

%run 'dbfs:/mnt/<<pathtopyfile>>' keeps reporting Notebook not found. Tried with and without dbfs and same error.

Re: How to import local python file in notebook?

vida — Mon, 11 Apr 2016 22:15:57 GMT

@Deepak Chokkadi %run doesn't take a dbfs path - it takes the path to the notebook from the workspace.

Re: How to import local python file in notebook?

dchokkadi1_5588 — Mon, 11 Apr 2016 22:28:54 GMT

Thanks @Vida Ha! Any other solution to execute a notebook from a mounted S3 path?

Re: How to import local python file in notebook?

jurmu — Fri, 13 Apr 2018 09:12:32 GMT

Replying here instead of raising a new question.

%run seems to be working OK, but what it does is to run the whole file which means that all the variables/functions become global. In this case the module cannot be used as an object "module.function" etc...

What would be a workaround to do that? I guess re-writing the module as a class could work? This just doesn't "feel" right, writing a class where it should just be a module.

Creating an egg has also been suggested, but again this would result in a package/library that cannot be easily edited.

Any thoughts?

Re: How to import local python file in notebook?

ThomasKastl — Tue, 19 Feb 2019 15:31:44 GMT

I have the exact same question. Databricks seems nice for some initial exploration / playing around but once it goes into professionalization being able to only use notebooks sets severe limits...

Re: How to import local python file in notebook?

amanpreetkaur — Fri, 03 May 2019 07:13:53 GMT

Is it possible to import a particular function using %run statement instead of running the whole notebook?

Re: How to import local python file in notebook?

Dipan — Thu, 18 Jul 2019 14:55:01 GMT

Databricks is a very rigid environment. They don't promote modularizing code. I don't understand why they disable / dissuade use of such basic concepts which are so generic to all programming languages.

Even after so many years this is still a problem

Re: How to import local python file in notebook?

JavierOrozco — Thu, 18 Jul 2019 22:18:38 GMT

The way to solve this problem is to add the path of your code to the system, then proceed to import modules selectively or all modules in a file.

You can download an example notebook from here https://github.com/javierorozco/databricks_import_python_module

import sys

// Add the path to system, local or mounted S3 bucket, e.g. /dbfs/mnt/<path_to_bucket> sys.path.append('/databricks/driver/') sys.path.append('/databricks/driver/databricks_import_python_module/') sys.path.append('/databricks/driver/databricks_import_python_module/test.py')

Re: How to import local python file in notebook?

shantanuneema — Wed, 25 Dec 2019 04:49:02 GMT

@javier.orozco@realeuesit.com, I was able to work with the file created in your repo (test.py) but my with my own modules I am getting error. Anything I am missing?

Re: How to import local python file in notebook?

awaiskaleem — Mon, 30 Dec 2019 11:20:01 GMT

Tried this. %run runs the py file, print a print statement in external file. But what I want is to get a variable from external file and use it in current notebook. That doesn't work. (added some print statements below, the variable in file is called pseg_main)