topic Databricks Asset Bundle artifacts with module out of the bundle root (sync path) in Data Engineering

Databricks Asset Bundle artifacts with module out of the bundle root (sync path)

Jorge3 — Wed, 13 Nov 2024 14:41:27 GMT

Hello everyone!
I’m currently working on a project with shared functionalities across different Databricks bundles. I have separate folders for each bundle, along with a common libs/ folder that holds some Python modules intended to be shared across bundles. My directory structure looks like this:

backend/ ├── libs/ │ ├── new_package/ │ │ ├── src/ │ │ │ └── new_package/ │ │ │ ├── __init__.py │ │ │ └── functions.py │ │ ├── tests/ │ │ ├── README.md │ │ ├── requirements.txt │ │ └── pyproject.toml │ │ │ └── second_package/ │ ├── src/ │ │ └── second_package/ │ │ ├── __init__.py │ │ └── functions.py │ ├── tests/ │ ├── README.md │ ├── requirements.txt │ └── pyproject.toml │ ├── bundle_1/ │ ├── jobs/ │ ├── tests/ │ ├── README.md │ ├── requirements.txt │ └── databricks.yml │ └── bundle_2/ ├── jobs/ ├── tests/ ├── README.md ├── requirements.txt └── databricks.yml

I’m using Databricks artifacts to build the wheel as per the documentation. Since my modules reside outside of the bundle root, I use the sync paths mapping to include files located outside the bundle root. However, when I try deploying from the CLI, I encounter this error:

Error: build failed default, error: chdir C:\Users\***\python_monorepo\bundle_1\libs\new_package: The system cannot find the path specified., output:

It seems that the bundle is unable to access the external module. Here’s my databricks.yml configuration:

bundle: name: deploy-package-job resources: jobs: deploy-package-job: name: deploy-package-job job_clusters: - job_cluster_key: common-cluster new_cluster: spark_version: 13.3.x-scala2.12 node_type_id: Standard_DS3_v2 num_workers: 1 tasks: - task_key: notebook-task job_cluster_key: common-cluster notebook_task: notebook_path: ./jobs/notebook.ipynb libraries: - whl: ../libs/dist/*.whl sync: paths: - ../libs artifacts: default: type: whl build: python -m build path: ../libs/new_package targets: dev: mode: development default: true workspace: host: https://*** prod: mode: production workspace: host: https://***

Does anyone have suggestions or insights on why the bundle can’t locate the external module? Any advice would be greatly appreciated!

Thank you!

Re: Databricks Asset Bundle artifacts with module out of the bundle root (sync path)

VZLA — Fri, 13 Dec 2024 15:59:13 GMT

Hi @Jorge3, where you able to get this issue resolved?

I believe your artifact build path points outside the synced directory structure, and after syncing ../libs, libs should be available within the bundle root, so the artifact path should be updated accordingly. For example:

sync: paths: - ../libs artifacts: default: type: whl build: python -m build path: libs/new_package

To further elaborate about this theory, the artifacts.path is using ../libs/new_package, which references a directory above your bundle’s root directory. Once the sync process maps ../libs into your bundle’s local environment, the synced files appear as a subdirectory (libs/) inside the bundle root. Since the build process runs relative to the bundle root, using ../libs/new_package tries to access a location outside the newly synced directory structure. By changing ../libs/new_package to libs/new_package, you point directly to the synced module directory now located inside the bundle’s environment, ensuring the build command can find and build the wheel without encountering a "path not found" error. Hope it helps.

Re: Databricks Asset Bundle artifacts with module out of the bundle root (sync path)

Jorge3 — Thu, 03 Apr 2025 13:07:26 GMT

Sorry for the late response, since I've abandoned that project since then.

Thanks for the answer @VZLA , but I've tried today the way you suggested and it's not working either. The error I got is:
Error: directory libs/new_versiro_package not found
I got the same error from the validation and the deploy commands