cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Is it possible to import functions from a module in Workspace/Shared instead of Repos?

Christine
Contributor

Hi,

I am considering creating libraries for my databricks notebooks, and found that it is possible to import functions from modules saved in repos. Is it possible to move the .py files with the functions to Workspace/Shared and still import functions as if it was a library? 

It is working for me when I work in the Repos section, but I get the error "ModuleNotFoundError: No module named <folder_name>" when I run the same script in the same folder structure under Workspace/Shared.

I have also tried to append the path with sys.path.append('Workspace/Shared/<repos name>/<folder_name>') but with no luck.

I have observed that in the Shared folder, the .py files are not visible. Is it possible to enable files in Shared as for Repos?

Thank you!

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

@Christine Pedersen​ :

Yes, it is possible to create libraries for your Databricks notebooks and import functions from modules saved in repos, as well as from modules stored in the Workspace/Shared directory. However, you will need to take a few extra steps to properly import the modules stored in the Shared directory.

Here are the steps to follow:

  1. Move the .py files containing the functions you want to import to the Workspace/Shared folder.
  2. Create an empty file called __init__.py in the same directory as your .py files. This is necessary to make Python recognize the directory as a package.
  3. In your notebook, add the following code at the beginning to add the Workspace/Shared folder to the Python path:
import sys
sys.path.append("/Workspace/Shared")

4) To import functions from a module, use the following syntax:

from <folder_name>.<module_name> import <function_name>

For example, if your folder name is my_lib and your module name is my_module, and you want to import the function my_function, you would use:

from my_lib.my_module import my_function

Note that you should omit the .py extension when specifying the module name.

By following these steps, you should be able to import functions from modules stored in the Workspace/Shared directory as if they were in a library.

View solution in original post

4 REPLIES 4

Anonymous
Not applicable

@Christine Pedersen​ :

Yes, it is possible to create libraries for your Databricks notebooks and import functions from modules saved in repos, as well as from modules stored in the Workspace/Shared directory. However, you will need to take a few extra steps to properly import the modules stored in the Shared directory.

Here are the steps to follow:

  1. Move the .py files containing the functions you want to import to the Workspace/Shared folder.
  2. Create an empty file called __init__.py in the same directory as your .py files. This is necessary to make Python recognize the directory as a package.
  3. In your notebook, add the following code at the beginning to add the Workspace/Shared folder to the Python path:
import sys
sys.path.append("/Workspace/Shared")

4) To import functions from a module, use the following syntax:

from <folder_name>.<module_name> import <function_name>

For example, if your folder name is my_lib and your module name is my_module, and you want to import the function my_function, you would use:

from my_lib.my_module import my_function

Note that you should omit the .py extension when specifying the module name.

By following these steps, you should be able to import functions from modules stored in the Workspace/Shared directory as if they were in a library.

Hi @Suteja Kanuri​,

Thank you very much, this worked perfectly!

psps
New Contributor III

@Suteja Kanuri​  I tried it on cluster with DBR 9.1 and it didn't work. Any prerequisites for this solution ?

Anonymous
Not applicable

Hi @Christine Pedersen​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.