Convert notebook to Python library

[adrianhernandez]

Looking for ways to convert a Databricks notebook to Python library. Some context :

1. Don't want to give execute permissions to shared notebooks as we want to hide code from users.
2. Proposed solution is to have our shared notebook converted into a Python library. Goal is to import this Python library so users can create their own notebooks.
3. Exploring the use of Python Wheels, but, these may be restricted in our environment (managed by another team with very strict security).
4. Also tried converting code to UDFs but you cannot reference any spark in your code as UDF must be fully Python (cannot mix and match PySpark and Spark SQL in the same function).
5. Besides wheels and library imports is there any other way where we can accomplish sharing code w/o users viewing the code?

Thanks.

[mark_ott]

The best way to share code from a Databricks notebook as a reusable module while hiding implementation details from users—without using wheels or granting direct notebook execution permissions—is to convert your notebook into a Python module, store it securely in the Databricks workspace, and import it using relative workspace paths. This method allows importing custom Python modules directly from workspace files, which users can use without seeing source code, as long as permissions are set appropriately.

Workspace Modules Approach

• Python source files (.py) can be uploaded and stored alongside notebooks in the Databricks workspace.

• These modules can be organized in folders, and notebooks can import functions/classes using Python’s standard import syntax and relative paths (e.g., from workspace.module import func).

• By assigning workspace permissions, users can only read and use the module interface; actual code can be hidden if the file/folder access is restricted.

• If workspace permissions are appropriately managed, this can hide the code implementation while exposing APIs for user notebooks to call.

Limitations of Wheel Files and Library Installs

• Python wheels are commonly used for sharing and deploying modules, but strict environment policies can limit their use.

• Library uploads (including wheels) are not always feasible due to administrative restrictions.

• Notebook-scoped libraries are also possible but may not meet your code privacy requirements—users might still see source.

Additional Strategies

• Code obfuscation/minification: Not ideal for Python, as bytecode isn’t very secure and users might still find ways to read code if they have access.

• Docker containers: You can deploy Spark code in containers, hiding source, but this requires cluster admin and more setup.

• Unity Catalog or secret management: These help protect sensitive data, but cannot fully hide code logic.

For your needs, using Python modules stored in workspace files with carefully set permissions is the most suitable way to share code without exposing internals—provided your environment supports this feature. Wheels and other binary approaches remain best but may be restricted. UDFs and notebook-scoped libraries do not fully solve the visibility and Spark referencing problems.

[adrianhernandez]

Thanks for the great information. Our team has decided to do this as a wheel. Can a notebook be created that pushes new versions of code w/o having to go thru the manual process of creating a whl and other configuration files? In other words, can I create a notebook that will setup/configure and install the wheel?