topic Re: How do I import a python module when deploying with DAB? in Data Engineering

How do I import a python module when deploying with DAB?

kenny_hero — Fri, 02 Jan 2026 23:33:38 GMT

Below is how the folder structure of my project looks like:

I created a python module, logger.py, that I want to reuse in many transformers, so I put it in utils folder. I want transformer_1.py to import get_logger() function from the module:

# transformer_1.py from pyspark import pipelines as dp from utils.logger import get_logger @dp.table def load_events(): logger = get_logger(__name__) logger.info('Loading events...') ...

After I deploy my project using DAB, running "etl_event" job fails because 'utils' module is not found.

I learned the following from researching the documents:

PYTHONPATH is not set to the root of the bundle.
One suggestion is to use `sys.path.append()` before each import. This feels rather a hack that I must remember to manipulate system path before each import.
Another suggestion is to build a python wheel file. I'm not sure if that applies to my project because the module(utils/logger.py) is already included in bundle deployed. I should not need to build a separate wheel file.

My question is what is the proper way to configure my project/bundle so that I can have reusable python modules to be used in pipeline transformers?

Re: How do I import a python module when deploying with DAB?

Hubert-Dudek — Sat, 03 Jan 2026 20:29:11 GMT

Under src I would add the parent directory for pipeline and utils and put there __init__.py

Re: How do I import a python module when deploying with DAB?

Sanjeeb2024 — Sun, 04 Jan 2026 06:05:58 GMT

Thank you @Hubert-Dudek . This is very useful.

Re: How do I import a python module when deploying with DAB?

kenny_hero — Mon, 05 Jan 2026 17:51:45 GMT

@Hubert-Dudek, thank you for your response. I really appreciate it.

However, I still cannot get the import to work even after I follow your instructions. Here is the folder structure:

The transformer code is below:

from pyspark import pipelines as dp from project.utils.logger import get_logger #from utils.logger import get_logger @dp.table def sample_users(): logger = get_logger(__name__) logger.info('Running sample_users transformer') return ( spark.read.table("samples.wanderbricks.users") .select("user_id", "email", "name", "user_type") )

After deploying the bundle, I tried running the job but it still fails to import the logger module.

Traceback (most recent call last): File "/Workspace/Users/.../.bundle/import_test/dev/files/src/project/pipelines/import_test/transformations/sample_users.py", cell 1, line 3 1 from pyspark import pipelines as dp ----> 3 from project.utils.logger import get_logger 4 #from utils.logger import get_logger 6 @dp.table 7 def sample_users(): File "/databricks/python_shell/lib/dbruntime/autoreload/discoverability/hook.py", line 71, in AutoreloadDiscoverabilityHook._patched_import(self, name, *args, **kwargs) 65 if not self._should_hint and ( 66 (module := sys.modules.get(absolute_name)) is not None and 67 (fname := get_allowed_file_name_or_none(module)) is not None and 68 (mtime := os.stat(fname).st_mtime) > self.last_mtime_by_modname.get( 69 absolute_name, float("inf")) and not self._should_hint): 70 self._should_hint = True ---> 71 module = self._original_builtins_import(name, *args, **kwargs) 72 if (fname := fname or get_allowed_file_name_or_none(module)) is not None: 73 mtime = mtime or os.stat(fname).st_mtime ModuleNotFoundError: No module named 'project' Error: [update_progress] Update 803be7 is FAILED.

I tried importing from both "project.utils.logger" and "utils.logger" and neither worked.

Can you help me figure out what I'm doing wrong?

Re: How do I import a python module when deploying with DAB?

rcdatabricks — Tue, 06 Jan 2026 17:53:27 GMT

you have to __init__.py has to be under src/__init__.py folder. looks like you created it under utils/__init__.py

Re: How do I import a python module when deploying with DAB?

kenny_hero — Tue, 06 Jan 2026 18:13:03 GMT

@rcdatabricks,

I tried that.

But import still fails.

ile "/Workspace/Users/.../.bundle/import_test/dev/files/src/project/pipelines/import_test/transformations/sample_users.py", cell 1, line 3 1 from pyspark import pipelines as dp ----> 3 from project.utils.logger import get_logger 4 # from utils.logger import get_logger 6 @dp.table 7 def sample_users(): File "/databricks/python_shell/lib/dbruntime/autoreload/discoverability/hook.py", line 71, in AutoreloadDiscoverabilityHook._patched_import(self, name, *args, **kwargs) 65 if not self._should_hint and ( 66 (module := sys.modules.get(absolute_name)) is not None and 67 (fname := get_allowed_file_name_or_none(module)) is not None and 68 (mtime := os.stat(fname).st_mtime) > self.last_mtime_by_modname.get( 69 absolute_name, float("inf")) and not self._should_hint): 70 self._should_hint = True ---> 71 module = self._original_builtins_import(name, *args, **kwargs) 72 if (fname := fname or get_allowed_file_name_or_none(module)) is not None: 73 mtime = mtime or os.stat(fname).st_mtime ModuleNotFoundError: No module named 'project'

I'm new to Databricks, so I may not fully understand how bundle deployment work. I hope someone can share knowledge to help me with a problem what I feel trivial.

Re: How do I import a python module when deploying with DAB?

kenny_hero — Thu, 22 Jan 2026 15:51:05 GMT

Just to share knowledge with other readers, I think the best option for now(until Databricks supports adding project root to PYTHONPATH) is to refactor the project to have custom modules in a separate folder, create a wheel when building DAB and declare as dependency of pipeline/job config.

I followed the instruction in this page, but struggled to make it work. It turned out dependency config is different if pipeline runs on serverless. Instead of using "libraries" as in the instruction , I have to use "environment".

resources: pipelines: <pipeline name> environment: dependencies: - <path to wheel file>

I wish the instruction included this example for serverless, or Databricks supports same configuration regardless of compute type.

Re: How do I import a python module when deploying with DAB?

pradeep_singh — Sat, 24 Jan 2026 13:35:42 GMT

You dont need to use wheel files . Use glob as the key instead of file - https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipelinelibraries
Here is the screenshot .