topic Re: Cannot import relative python paths in Data Engineering

Cannot import relative python paths

georgef — Mon, 06 May 2024 14:18:19 GMT

Hello,

Some variations of this question have been asked before but there doesn't seem to be an answer for the following simple use case:

I have the following file structure on a Databricks Asset Bundles project:

src --dir1 ----file1.py --dir2 ----file2.py

There doesn't seem to be a way to import anything from file1.py to file2.py. I have tried (inside file2.py):

from src.dir1.file1 import something from dir1.file1 import something from ..dir1.file1 import something

None of these seem to work. I have tried appending os.abspath("..") and "../.." to sys.path, as well as passing ${workspace.file_path} to the task and appending that, to no avail.

Note that relative imports on the same level work so the following both work:

# src # --file1.py # --file2.py # We are on file1.py from file2 import something # src # --file1.py # --dir2 # ----file2.py # We are on file1.py from dir2.file2 import something

Is something weird going on with the import paths when deploying DABs? I'd rather avoid having to deploy a full-on wheel if I can avoid it, as it adds a lot of unnecessary boilerplate.

Thank you!

Re: Cannot import relative python paths

georgef — Fri, 02 Aug 2024 13:57:35 GMT

Unfortunately none of the methods proposed here worked, so we ended up having to wrap our code in packages and installing those in the cluster before running. Not an ideal workflow, but works for projects that are easily decoupled from their orchestration.

Re: Cannot import relative python paths

m997al — Fri, 02 Aug 2024 14:11:00 GMT

Hi. This was a long-standing issue for me too. This solution may not be what is desired, but it works perfectly for my needs.

In my python code, I have this structure:

if __name__ == '__main__': # directory structure where "mycode" is this code here, trying to import "mymodule" # > main # -- mymodule # > subdir # -- mycode # Get the current directory current_directory = os.getcwd() # Get the parent directory parent_directory = os.path.abspath(os.path.join(current_directory, '..')) # Add the parent directory to sys.path sys.path.append(parent_directory) from mymodule import myfunction

Re: Cannot import relative python paths

klaas — Thu, 10 Apr 2025 14:36:15 GMT

This works as long as the script calling the module is indeed __main__; i've changed it a bit to make it more generic:

import os import sys def find_module(path): while path: if os.path.basename(path) == "src": return path new_path = os.path.abspath(os.path.join(path, '..')) if new_path == path: # Prevent infinite loop break path = new_path return None # Start from the current directory current_directory = os.getcwd() # Find the source directory source_directory = find_module(current_directory) if source_directory: sys.path.append(source_directory) from src.core import MyModule else: raise ImportError("Databricks messed up my filepaths. Please complain somewhere.")