mark_ott
Databricks Employee
Databricks Employee

‎11-07-2025 08:34 AM

You are correctly following Databricks’ recommendation to store shared code in Python files and import them into your notebooks, especially for Delta Live Tables (DLT) pipelines and serverless environments. However, import path issues are common, particularly with DLT notebooks, Git folders, and workspace files.

Here’s why your import might not be working and how to resolve it:

Key Points from Documentation

  • Workspace Files & Git Folders:

    • If your .py file is not in the same directory as the notebook: you must manually append its location to sys.path, and if the file is coming from a Git folder, prepend /Workspace/ to the path.

    • If your .py file is in the same directory as the notebook, you should not need to adjust sys.path—the import should work directly.

Common Issues & Solutions

1. Verify Actual Pathing

The file location must be exact:

  • If in a Git folder: ensure you’re using /Workspace/RepoName/.../file.py (not just /RepoName/.../file.py).

  • If in a workspace directory: ensure the full path is /Workspace/.../file.py.

You can use a cell to list files and confirm the location:

python
import os print(os.listdir("/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib/"))

If your file isn’t listed, re-upload or move it.

2. Set sys.path Explicitly

If importing from outside the current notebook directory (e.g., from another folder or the root of a Git repo), use:

python
import sys sys.path.append('/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib') from time_and_date_funclib import *

For Git folder:

python
# If your notebook is at /Workspace/Repos/your.name@company.com/project/... sys.path.append('/Workspace/Repos/your.name@company.com/project/path_to_lib_folder') from time_and_date_funclib import *

Make sure /Workspace is prepended as documented.

3. Python File Naming

  • Ensure your filename is exactly time_and_date_funclib.py (case-sensitive, no typos).

  • The file must not be empty and should have valid Python code.

4. Notebook Path vs. DLT Pipeline Path

  • When running in DLT pipelines/serverless, Databricks may use a different working directory, so explicitly setting sys.path is often required—even when following the documentation.

Troubleshooting Checklist

  • Restart the cluster after making path changes/directory moves.

  • Re-upload the library and re-run the path listing cell to confirm the file exists.

  • Test import in a new notebook in the same location to rule out notebook path confusion.

  • Check Python version and workspace settings to ensure compatibility.

Example Working Pattern

python
import sys sys.path.append('/Workspace/your_full_path_to_lib') from time_and_date_funclib import *

Databricks DLT Pipelines Extra Notes

  • For production DLT pipeline jobs, consider packaging your library as a Wheel (whl) file, uploading it as a cluster library, or managing project dependencies with a workspace-specific solution for more robust imports.

  • If you're working inconsistently between interactive notebooks and DLT pipelines, paths might resolve differently—explicitly print and check sys.path in each environment.

References

In summary: Always prepend /Workspace/ for absolute paths, verify your file location and name, and update sys.path if not in the notebook's directory. For DLT/serverless, manage your project dependencies carefully—consider wheels or workspace libraries for consistent behavior across environments.

0 Kudos