ModuleNotFoundError Importing fuction modules to DLT pipelines

TamD — Tue, 18 Mar 2025 23:04:29 GMT

Following best practice, we want to avoid reusing code by putting commonly used transformations into function libraries and then importing and calling those functions where required.

We also want to follow Databricks recommendations to use serverless compute and DLT pipelines. However, I'm unable to successfully find a way to import our function libraries.

I’m following the instructions here:

https://docs.databricks.com/aws/en/dlt/import-workspace-files

I’ve created a file called time_and_date_funclib.py to store commonly used date and time transforms.

According to the docco linked above, I should be able to import the functions to my working notebook using the code:

from time_and_date_funclib import *

However, I get a ModuleNotFoundError:

ModuleNotFoundError: Traceback (most recent call last):
File "/Repos/tam.duran@flickelectric.co.nz/data_lakehouse/Data Engineering/30_silver_to_gold/consumption_data/network_banded_usage", cell 5, line 7
   4 import sys, os
   5 # sys.path.append(os.path.abspath("/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib/time_and_date_funclib"))
----> 7 from time_and_date_funclib import *
ModuleNotFoundError: No module named 'time_and_date_funclib'

I get this error even if time_and_date_funclib.py is stored in the same location as the calling notebook (in case I was formatting the absolute path incorrectly). I have also tried storing it at the root of the Git folder, per the note below (copied from the docco linked above).

note

If your notebook imports modules or packages from a workspace files path or a Git folders path different from the notebook directory, you must manually append the path to the files using sys.path.append().

If you are importing a file from a Git folder, you must prepend /Workspace/ to the path. For example, sys.path.append('/Workspace/...'). Omitting /Workspace/ from the path results in an error.

If the modules or packages are stored in the same directory as the notebook, you do not need to append the path manually. You also do not need to manually append the path when importing from the root directory of a Git folder because the root directory is automatically appended to the path.

Has anyone got this working? What am I missing here?

Re: ModuleNotFoundError Importing fuction modules to DLT pipelines

mark_ott — Fri, 07 Nov 2025 16:34:05 GMT

You are correctly following Databricks’ recommendation to store shared code in Python files and import them into your notebooks, especially for Delta Live Tables (DLT) pipelines and serverless environments. However, import path issues are common, particularly with DLT notebooks, Git folders, and workspace files.

Here’s why your import might not be working and how to resolve it:

Key Points from Documentation

Workspace Files & Git Folders:
- If your .py file is not in the same directory as the notebook: you must manually append its location to sys.path, and if the file is coming from a Git folder, prepend /Workspace/ to the path.
- If your .py file is in the same directory as the notebook, you should not need to adjust sys.path—the import should work directly.

Common Issues & Solutions

1. Verify Actual Pathing

The file location must be exact:

If in a Git folder: ensure you’re using /Workspace/RepoName/.../file.py (not just /RepoName/.../file.py).
If in a workspace directory: ensure the full path is /Workspace/.../file.py.

You can use a cell to list files and confirm the location:

python

import os
print(os.listdir("/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib/"))

If your file isn’t listed, re-upload or move it.

2. Set `sys.path` Explicitly

If importing from outside the current notebook directory (e.g., from another folder or the root of a Git repo), use:

python

import sys
sys.path.append('/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib')
from time_and_date_funclib import *

For Git folder:

python

# If your notebook is at /Workspace/Repos/your.name@company.com/project/...
sys.path.append('/Workspace/Repos/your.name@company.com/project/path_to_lib_folder')
from time_and_date_funclib import *

Make sure /Workspace is prepended as documented.

3. Python File Naming

Ensure your filename is exactly time_and_date_funclib.py (case-sensitive, no typos).
The file must not be empty and should have valid Python code.

4. Notebook Path vs. DLT Pipeline Path

When running in DLT pipelines/serverless, Databricks may use a different working directory, so explicitly setting sys.path is often required—even when following the documentation.

Troubleshooting Checklist

Restart the cluster after making path changes/directory moves.
Re-upload the library and re-run the path listing cell to confirm the file exists.
Test import in a new notebook in the same location to rule out notebook path confusion.
Check Python version and workspace settings to ensure compatibility.

Example Working Pattern

python

import sys
sys.path.append('/Workspace/your_full_path_to_lib')
from time_and_date_funclib import *

Databricks DLT Pipelines Extra Notes

For production DLT pipeline jobs, consider packaging your library as a Wheel (whl) file, uploading it as a cluster library, or managing project dependencies with a workspace-specific solution for more robust imports.
If you're working inconsistently between interactive notebooks and DLT pipelines, paths might resolve differently—explicitly print and check sys.path in each environment.

References

Databricks documentation: Import workspace files into notebooks and jobs

In summary: Always prepend /Workspace/ for absolute paths, verify your file location and name, and update sys.path if not in the notebook's directory. For DLT/serverless, manage your project dependencies carefully—consider wheels or workspace libraries for consistent behavior across environments.

topic Re: ModuleNotFoundError Importing fuction modules to DLT pipelines in Data Engineering

ModuleNotFoundError Importing fuction modules to DLT pipelines

Re: ModuleNotFoundError Importing fuction modules to DLT pipelines

Key Points from Documentation

Common Issues & Solutions

1. Verify Actual Pathing

2. Set sys.path Explicitly

3. Python File Naming

4. Notebook Path vs. DLT Pipeline Path

Troubleshooting Checklist

Example Working Pattern

Databricks DLT Pipelines Extra Notes

References

2. Set `sys.path` Explicitly