Databricks Asset Bundle artifacts with module out of the bundle root (sync path)

[Jorge3]

Hello everyone!
I’m currently working on a project with shared functionalities across different Databricks bundles. I have separate folders for each bundle, along with a common libs/ folder that holds some Python modules intended to be shared across bundles. My directory structure looks like this:

backend/
├── libs/
│   ├── new_package/
│   │   ├── src/
│   │   │   └── new_package/
│   │   │       ├── __init__.py
│   │   │       └── functions.py
│   │   ├── tests/
│   │   ├── README.md 
│   │   ├── requirements.txt    
│   │   └── pyproject.toml
│   │
│   └── second_package/
│       ├── src/
│       │   └── second_package/
│       │       ├── __init__.py
│       │       └── functions.py
│       ├── tests/
│       ├── README.md 
│       ├── requirements.txt    
│       └── pyproject.toml
│
├── bundle_1/
│   ├── jobs/
│   ├── tests/
│   ├── README.md
│   ├── requirements.txt
│   └── databricks.yml
│
└── bundle_2/
    ├── jobs/
    ├── tests/
    ├── README.md
    ├── requirements.txt
    └── databricks.yml

I’m using Databricks artifacts to build the wheel as per the documentation. Since my modules reside outside of the bundle root, I use the sync paths mapping to include files located outside the bundle root. However, when I try deploying from the CLI, I encounter this error:

Error: build failed default, error: chdir C:\Users\***\python_monorepo\bundle_1\libs\new_package: The system cannot find the path specified., output:

It seems that the bundle is unable to access the external module. Here’s my databricks.yml configuration:

bundle:
  name: deploy-package-job
resources:
  jobs:
    deploy-package-job: 
      name: deploy-package-job
      job_clusters:
        - job_cluster_key: common-cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: Standard_DS3_v2
            num_workers: 1
      tasks:
        - task_key: notebook-task
          job_cluster_key: common-cluster
          notebook_task:
            notebook_path: ./jobs/notebook.ipynb
          libraries:
            - whl: ../libs/dist/*.whl
sync:
  paths:
    - ../libs
artifacts:
  default:
    type: whl
    build: python -m build
    path: ../libs/new_package

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://***

  prod:
    mode: production
    workspace:
      host: https://***

Does anyone have suggestions or insights on why the bundle can’t locate the external module? Any advice would be greatly appreciated!

Thank you!

[VZLA]

Hi @Jorge3, where you able to get this issue resolved?

I believe your artifact build path points outside the synced directory structure, and after syncing ../libs, libs should be available within the bundle root, so the artifact path should be updated accordingly. For example:

sync:
  paths:
    - ../libs

artifacts:
  default:
    type: whl
    build: python -m build
    path: libs/new_package

To further elaborate about this theory, the artifacts.path is using ../libs/new_package, which references a directory above your bundle’s root directory. Once the sync process maps ../libs into your bundle’s local environment, the synced files appear as a subdirectory (libs/) inside the bundle root. Since the build process runs relative to the bundle root, using ../libs/new_package tries to access a location outside the newly synced directory structure. By changing ../libs/new_package to libs/new_package, you point directly to the synced module directory now located inside the bundle’s environment, ensuring the build command can find and build the wheel without encountering a "path not found" error. Hope it helps.