Cannot import relative python paths

[georgef]

Hello,

Some variations of this question have been asked before but there doesn't seem to be an answer for the following simple use case:

I have the following file structure on a Databricks Asset Bundles project:

src
--dir1
----file1.py
--dir2
----file2.py

There doesn't seem to be a way to import anything from file1.py to file2.py. I have tried (inside file2.py):

from src.dir1.file1 import something
from dir1.file1 import something
from ..dir1.file1 import something

None of these seem to work. I have tried appending os.abspath("..") and "../.." to sys.path, as well as passing ${workspace.file_path} to the task and appending that, to no avail.

Note that relative imports on the same level work so the following both work:

# src
# --file1.py
# --file2.py
# We are on file1.py
from file2 import something

# src
# --file1.py
# --dir2
# ----file2.py
# We are on file1.py
from dir2.file2 import something

Is something weird going on with the import paths when deploying DABs? I'd rather avoid having to deploy a full-on wheel if I can avoid it, as it adds a lot of unnecessary boilerplate.

Thank you!

[Kaniz_Fatma]

Hi @georgef, It appears that you’re encountering issues with importing modules within a Databricks Asset Bundles (DABs) project.

Let’s explore some potential solutions to address this problem.

1. Bundle Deployment and Import Paths:

   • When deploying a DABs project, the Python import paths can behave differently compared to local development.
   • DABs bundles are designed to package and distribute code, but they may not handle relative imports in the same way as local Python environments.
   • To address this, consider the following approaches:

2. Explicitly Add the Source Directory to sys.path:

   • In your file2.py, you can explicitly add the source directory (src) to the Python path using sys.path.append().
   • Modify file2.py as follows:

   import sys
   sys.path.append('/Workspace/${workspace.file_path}/src')
   from src.dir1.file1 import something
   

   • Replace ${workspace.file_path} with the actual path to your workspace. This approach ensures that the src directory is included in the Python path.

3. Use Parameters in Databricks YAML:

   • In your databricks.yml file, pass an argument to your script via parameters.
   • Update your databricks.yml as follows:

   resources:
     jobs:
       my_job:
         name: my_job
         tasks:
           - task_key: my_task
             existing_cluster_id: YYY
             spark_python_task:
               python_file: src/jobs/bronze/my_script.py
               parameters: ['/Workspace/${workspace.file_path}/src']
   

   • In my_script.py, retrieve the parameter and add it to the Python path:

   import sys
   bundle_src_path = sys.argv[1]
   sys.path.append(bundle_src_path)
   from src.jobs.common import *
   

   • Again, replace ${workspace.file_path} with the actual path to your workspace.

4. Other Approaches:

   • If the above solutions don’t work, consider alternative deployment methods:
     • Bundle and deploy your code as a library on the cluster.
     • Pip install the Git repository directly (requires credentials in secrets manager).
     • Use %run with inline code in a notebook.
     • Use a custom container image¹.

Remember to adjust the paths and configurations according to your specific setup. DABs can be powerful for packaging and distributing code, but understanding their behaviour is crucial for successful deployment.

Good luck! 😊