topic Re: Inconsistent PYTHONPATH, Git folders vs DAB in Data Engineering

Inconsistent PYTHONPATH, Git folders vs DAB

mydefaultlogin — Mon, 24 Mar 2025 14:28:39 GMT

Hello Databricks Community,

I'm encountering an issue related to Python paths when working with notebooks in Databricks. I have a following structure in my project:

my_notebooks - my_notebook.py /my_package - __init__.py - hello.py databricks.yml

my_notebook.py

# Databricks notebook source import sys print(*sys.path, sep='\n') # COMMAND ---------- from my_package.hello import hello_world hello_world()

hello.py:

def hello_world(): print('Hello World')

databricks.yml

bundle: name: my_bundle targets: dev: mode: production workspace: host: https://adb-XXXXXXXXXXXXXXXXXXXXX.azuredatabricks.net root_path: /dev/${bundle.name}

I would like to work with the notebooks in two ways:
- I want all developers to work in their own Git folders
- then, once their work is done, and a pull request is completed, a pipeline would call `databricks bundle deploy` command to deploy the code to the `dev` environment.

The problem is that when you execute a notebook from someone's Git folder it works fine because Python easily finds the `my_package` package. When you try to execute `my_notebook` from the location it is deployed to by DAB (`/Workspace/dev/my_bundle/files/my_notebooks/my_notebook`), it does not work because `my_package` cannot be imported.

What is the reason for this inconsistency?
I would like to be able to import python packages from the root of my project, and the fact that the python path behaviour is different in git folders and DAB.

Re: Inconsistent PYTHONPATH, Git folders vs DAB

Brahmareddy — Mon, 24 Mar 2025 20:31:06 GMT

Hi mydefaultlogin,

How are you doing today?, As per my understanding, You're right—this happens because when you're running notebooks from your Git folder, Python knows exactly where your project root is and can easily find my_package. But when you deploy using Databricks Asset Bundles (DAB), the notebook runs from a different path (like /Workspace/dev/...), and Python no longer sees your project root by default, so it can't find the package. A simple fix is to manually add your project root to Python's path at the top of your notebook using sys.path.append('/Workspace/dev/my_bundle/files'). This helps Python locate your package just like it does in the Git setup. It’s a common issue with DAB deployments, and this quick adjustment should solve it. Let me know if you’d like help making it reusable!

Regards,

Brahma

Re: Inconsistent PYTHONPATH, Git folders vs DAB

kenny_hero — Tue, 30 Dec 2025 19:12:14 GMT

I have a related question.

I'm new to Databricks platform. I struggle with PYTHONPATH issue as the original poster raised. I understand using sys.path.append(...) is one approach for notebook. This is acceptable for ad-hoc interactive session, but this doesn't feel right for python module. Suppose I have multiple python modules that need to import a custom module, like the hello.py in the OP's example. I must have sys.path.append(...) line in every module before I import hello.py?

To give an example, let's say I have an utility python module that's a base of other custom modules. Then I have to remember to do sys.path.append(...) before every import. Worse part is if I forget sys.path.append(...), import works in local dev(because most IDE provides a way to set PYTHONPATH to project root) but import fails only after I deploy bundle later.

I feel there must be a way to ensure consistent behavior programmatically without explicit path setting.