Is it possible to import functions from a module in Workspace/Shared instead of Repos?

[Christine]

Hi,

I am considering creating libraries for my databricks notebooks, and found that it is possible to import functions from modules saved in repos. Is it possible to move the .py files with the functions to Workspace/Shared and still import functions as if it was a library? 

It is working for me when I work in the Repos section, but I get the error "ModuleNotFoundError: No module named <folder_name>" when I run the same script in the same folder structure under Workspace/Shared.

I have also tried to append the path with sys.path.append('Workspace/Shared/<repos name>/<folder_name>') but with no luck.

I have observed that in the Shared folder, the .py files are not visible. Is it possible to enable files in Shared as for Repos?

Thank you!

[Anonymous · Accepted solution]

@Christine Pedersen​ :

Yes, it is possible to create libraries for your Databricks notebooks and import functions from modules saved in repos, as well as from modules stored in the Workspace/Shared directory. However, you will need to take a few extra steps to properly import the modules stored in the Shared directory.

Here are the steps to follow:

1. Move the .py files containing the functions you want to import to the Workspace/Shared folder.
2. Create an empty file called __init__.py in the same directory as your .py files. This is necessary to make Python recognize the directory as a package.
3. In your notebook, add the following code at the beginning to add the Workspace/Shared folder to the Python path:

import sys
sys.path.append("/Workspace/Shared")

4) To import functions from a module, use the following syntax:

from <folder_name>.<module_name> import <function_name>

For example, if your folder name is my_lib and your module name is my_module, and you want to import the function my_function, you would use:

from my_lib.my_module import my_function

Note that you should omit the .py extension when specifying the module name.

By following these steps, you should be able to import functions from modules stored in the Workspace/Shared directory as if they were in a library.

[Christine]

Hi @Suteja Kanuri​,

Thank you very much, this worked perfectly!

[psps]

@Suteja Kanuri​  I tried it on cluster with DBR 9.1 and it didn't work. Any prerequisites for this solution ?

[Anonymous]

Hi @Christine Pedersen​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!