Cannot import relative python paths

[georgef]

Hello,

Some variations of this question have been asked before but there doesn't seem to be an answer for the following simple use case:

I have the following file structure on a Databricks Asset Bundles project:

src
--dir1
----file1.py
--dir2
----file2.py

There doesn't seem to be a way to import anything from file1.py to file2.py. I have tried (inside file2.py):

from src.dir1.file1 import something
from dir1.file1 import something
from ..dir1.file1 import something

None of these seem to work. I have tried appending os.abspath("..") and "../.." to sys.path, as well as passing ${workspace.file_path} to the task and appending that, to no avail.

Note that relative imports on the same level work so the following both work:

# src
# --file1.py
# --file2.py
# We are on file1.py
from file2 import something

# src
# --file1.py
# --dir2
# ----file2.py
# We are on file1.py
from dir2.file2 import something

Is something weird going on with the import paths when deploying DABs? I'd rather avoid having to deploy a full-on wheel if I can avoid it, as it adds a lot of unnecessary boilerplate.

Thank you!

[Kaniz_Fatma]

Hi @georgef, It appears that you’re encountering issues with importing modules within a Databricks Asset Bundles (DABs) project.

Let’s explore some potential solutions to address this problem.

1. Bundle Deployment and Import Paths:

   • When deploying a DABs project, the Python import paths can behave differently compared to local development.
   • DABs bundles are designed to package and distribute code, but they may not handle relative imports in the same way as local Python environments.
   • To address this, consider the following approaches:

2. Explicitly Add the Source Directory to sys.path:

   • In your file2.py, you can explicitly add the source directory (src) to the Python path using sys.path.append().
   • Modify file2.py as follows:

   import sys
   sys.path.append('/Workspace/${workspace.file_path}/src')
   from src.dir1.file1 import something
   

   • Replace ${workspace.file_path} with the actual path to your workspace. This approach ensures that the src directory is included in the Python path.

3. Use Parameters in Databricks YAML:

   • In your databricks.yml file, pass an argument to your script via parameters.
   • Update your databricks.yml as follows:

   resources:
     jobs:
       my_job:
         name: my_job
         tasks:
           - task_key: my_task
             existing_cluster_id: YYY
             spark_python_task:
               python_file: src/jobs/bronze/my_script.py
               parameters: ['/Workspace/${workspace.file_path}/src']
   

   • In my_script.py, retrieve the parameter and add it to the Python path:

   import sys
   bundle_src_path = sys.argv[1]
   sys.path.append(bundle_src_path)
   from src.jobs.common import *
   

   • Again, replace ${workspace.file_path} with the actual path to your workspace.

4. Other Approaches:

   • If the above solutions don’t work, consider alternative deployment methods:
     • Bundle and deploy your code as a library on the cluster.
     • Pip install the Git repository directly (requires credentials in secrets manager).
     • Use %run with inline code in a notebook.
     • Use a custom container image¹.

Remember to adjust the paths and configurations according to your specific setup. DABs can be powerful for packaging and distributing code, but understanding their behaviour is crucial for successful deployment.

Good luck! 😊

[georgef · Accepted solution]

Unfortunately none of the methods proposed here worked, so we ended up having to wrap our code in packages and installing those in the cluster before running. Not an ideal workflow, but works for projects that are easily decoupled from their orchestration.

[m997al]

Hi.  This was a long-standing issue for me too.  This solution may not be what is desired, but it works perfectly for my needs.

In my python code, I have this structure:

if __name__ == '__main__':

    # directory structure where "mycode" is this code here, trying to import "mymodule"
    #  > main
    #    -- mymodule
    #    > subdir
    #      -- mycode

    # Get the current directory
    current_directory = os.getcwd()

    # Get the parent directory
    parent_directory = os.path.abspath(os.path.join(current_directory, '..'))

    # Add the parent directory to sys.path
    sys.path.append(parent_directory)

    from mymodule import myfunction