topic Re: Importing python module in Data Engineering

Importing python module

tariq — Tue, 13 Sep 2022 14:46:38 GMT

I'm not sure how a simple thing like importing a module in python can be so broken in such a product. First, I was able to make it work using the following:

import sys
sys.path.append("/Workspace/Repos/Github Repo/sparkling-to-databricks/src")
from utils.some_util import *

I was able to use the imported function. But then I restarted the cluster and this would not work even though the path is in sys.path.

I also tried the following:

spark.sparkContext.addPyFile("/Workspace/Repos/Github Repo/my-repo/src/utils/some_util.py")

This did not work either. Can someone please tell me what I'm doing wrong here and suggest a solution. Thanks.

Re: Importing python module

Anonymous — Tue, 13 Sep 2022 14:58:50 GMT

https://docs.databricks.com/libraries/index.html the docs should help.

Keep in mind that paths on the cluster are distributed paths and that paths in python are local to the driver. If you do %fs ls / you'll get a different result than if you do %sh fs ls /

Re: Importing python module

tariq — Tue, 13 Sep 2022 15:09:36 GMT

So there's no other way than creating a library?

Re: Importing python module

KrishZ — Wed, 14 Sep 2022 04:36:54 GMT

I too wonder the same thing. How can importing a python module be so difficult and not even documented lol.

No need for libraries..

Here's what worked for me..

Step1: Upload the module by first opening a notebook >> File >> Upload Data >> drag and drop your module

Step2: Click on Next

Step3: Copy the databricks path for your module. (this path is diplayed in the pop up that you see just after click on Next)

For me , if my module is named test_module the path looks like

dbfs:/FileStore/shared_uploads/krishz@company.com/test_module.py

Step4: Append the above to Path (albeit with changes)

Change 1: Change dbfs:/ to /dbfs/
Change 2: remove your module name from the path 🙂

Now my path to append looks like

/dbfs/FileStore/shared_uploads/krishz@company.com

Step5:

import sys
sys.path.append("/dbfs/FileStore/shared_uploads/krishz@company.com")

Step6:

Now you can import simply by using below:

import test_module

Post this, you may write code as if you had imported test_module in your usual Jupyter notebook - No need to worry about any databricks intricacies.

Lemme know if you are unclear about any step. Honestly speaking, I don't know why you were recommended to use libraries for such a simple request.

Re: Importing python module

tariq — Wed, 14 Sep 2022 05:00:22 GMT

Thanks for the reply. The file I have is part of a repo within the repo structure. Is there a way to import dependencies within a repo. The repo structure looks something like below:

And I need to import the some_util module in my_notebook.