ModuleNotFoundError Importing fuction modules to DLT pipelines

[TamD]

Following best practice, we want to avoid reusing code by putting commonly used transformations into function libraries and then importing and calling those functions where required.

We also want to follow Databricks recommendations to use serverless compute and DLT pipelines. However, I'm unable to successfully find a way to import our function libraries.

I’m following the instructions here:

https://docs.databricks.com/aws/en/dlt/import-workspace-files

I’ve created a file called time_and_date_funclib.py to store commonly used date and time transforms.

According to the docco linked above, I should be able to import the functions to my working notebook using the code:

from time_and_date_funclib import *

However, I get a ModuleNotFoundError:

ModuleNotFoundError: Traceback (most recent call last):
File "/Repos/tam.duran@flickelectric.co.nz/data_lakehouse/Data Engineering/30_silver_to_gold/consumption_data/network_banded_usage", cell 5, line 7
   4 import sys, os
   5 # sys.path.append(os.path.abspath("/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib/time_and_date_funclib"))
----> 7 from time_and_date_funclib import *
ModuleNotFoundError: No module named 'time_and_date_funclib'

I get this error even if time_and_date_funclib.py is stored in the same location as the calling notebook (in case I was formatting the absolute path incorrectly).  I have also tried storing it at the root of the Git folder, per the note below (copied from the docco linked above).

note

If your notebook imports modules or packages from a workspace files path or a Git folders path different from the notebook directory, you must manually append the path to the files using sys.path.append().

If you are importing a file from a Git folder, you must prepend /Workspace/ to the path. For example, sys.path.append('/Workspace/...'). Omitting /Workspace/ from the path results in an error.

If the modules or packages are stored in the same directory as the notebook, you do not need to append the path manually. You also do not need to manually append the path when importing from the root directory of a Git folder because the root directory is automatically appended to the path.

 Has anyone got this working?  What am I missing here?

[mark_ott]

You are correctly following Databricks’ recommendation to store shared code in Python files and import them into your notebooks, especially for Delta Live Tables (DLT) pipelines and serverless environments. However, import path issues are common, particularly with DLT notebooks, Git folders, and workspace files.

Here’s why your import might not be working and how to resolve it:

Key Points from Documentation

• Workspace Files & Git Folders:

  • If your .py file is not in the same directory as the notebook: you must manually append its location to sys.path, and if the file is coming from a Git folder, prepend /Workspace/ to the path.

  • If your .py file is in the same directory as the notebook, you should not need to adjust sys.path—the import should work directly.

Common Issues & Solutions

1. Verify Actual Pathing

The file location must be exact:

• If in a Git folder: ensure you’re using /Workspace/RepoName/.../file.py (not just /RepoName/.../file.py).

• If in a workspace directory: ensure the full path is /Workspace/.../file.py.

You can use a cell to list files and confirm the location:

python

import os print(os.listdir("/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib/"))

If your file isn’t listed, re-upload or move it.

2. Set sys.path Explicitly

If importing from outside the current notebook directory (e.g., from another folder or the root of a Git repo), use:

python

import sys sys.path.append('/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib') from time_and_date_funclib import *

For Git folder:

python

# If your notebook is at /Workspace/Repos/your.name@company.com/project/... sys.path.append('/Workspace/Repos/your.name@company.com/project/path_to_lib_folder') from time_and_date_funclib import *

Make sure /Workspace is prepended as documented.

3. Python File Naming

• Ensure your filename is exactly time_and_date_funclib.py (case-sensitive, no typos).

• The file must not be empty and should have valid Python code.

4. Notebook Path vs. DLT Pipeline Path

• When running in DLT pipelines/serverless, Databricks may use a different working directory, so explicitly setting sys.path is often required—even when following the documentation.

Troubleshooting Checklist

• Restart the cluster after making path changes/directory moves.

• Re-upload the library and re-run the path listing cell to confirm the file exists.

• Test import in a new notebook in the same location to rule out notebook path confusion.

• Check Python version and workspace settings to ensure compatibility.

Example Working Pattern

python

import sys sys.path.append('/Workspace/your_full_path_to_lib') from time_and_date_funclib import *

Databricks DLT Pipelines Extra Notes

• For production DLT pipeline jobs, consider packaging your library as a Wheel (whl) file, uploading it as a cluster library, or managing project dependencies with a workspace-specific solution for more robust imports.

• If you're working inconsistently between interactive notebooks and DLT pipelines, paths might resolve differently—explicitly print and check sys.path in each environment.

---

References

• Databricks documentation: Import workspace files into notebooks and jobs

---

In summary: Always prepend /Workspace/ for absolute paths, verify your file location and name, and update sys.path if not in the notebook's directory. For DLT/serverless, manage your project dependencies carefully—consider wheels or workspace libraries for consistent behavior across environments.