I can't use my own .whl package in Databricks app with databricks asset bundles

[Oumeima]

I am building a databricks app using databricks asset bundles. I need to use a helpers packages that i built as an artifact and using in other resources outside the app. The only way to use it is to have the built package inside the app source code folder and reference it in my requirements.txt file.

But the issue is it has to be tracked in my github repo and i want to avoid that.

Any idea on how to do it ? I'd appreciate the help!

Thank you!

[stbjelcevic · Accepted solution]

Hi @Oumeima ,

One potential way around this is to upload the wheel file into a Unity Catalog volume or workspace file.

For the volume route, reference it directly in your app’s requirements.txt using an absolute /Volumes/<catalog>/<schema>/<volume>/...whl path. This is a supported and secure pattern for Databricks Apps dependency installation and avoids tracking the wheel in your app repo (source).

The workspace file route is very similar to the volume route. Specify the full path, e.g. /Workspace/Shared/Libraries/my-wheel-0.0.1-py3-none-any.whl in your databricks.yml file (source).

Those are probably the 2 easiest routes to achieve what you are trying to do. Please note that you will need to make sure the wheel file is updated if you make changes to the source code, since it will live outside of CI/CD.

Let me know if neither of the above works for you and I can try to come up with another solution.

[Oumeima]

Hello, thank you for the reply. That's precisely what i did actually. I built a CI/CD pipeline to deploy the package to a volume and then referenced it in my requirements.txt file.

[nk-five1]

Hi @Oumeima,
how did you manage to install your package in the app? Did you have to employ any tricks to get the volume to show up in the app?
Currently, I am trying the same but installation fails whatever I try because the mount point for the volume inside the app does not exist.

ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/Volumes/catalog/schema/dependencies/package_name-1.0.0-py3-none-any.whl'
Processing /Volumes/catalog/schema/dependencies/package_name-1.0.0-py3-none-any.whl (from -r requirements.txt (line 50))
WARNING: Requirement '/Volumes/catalog/schema/dependencies/package_name-1.0.0-py3-none-any.whl' looks like a filename, but the file does not exist

Thank you!

[Oumeima]

Hello,

The solution i found is as follows:

In my CI/CD pipeline that takes care of deploying the databricks app (am using databricks asset bundles to manage my resources), i added a job running a shell script that appends the volume path to the requirements.txt file before running the deployment job (very important). If you have multiple environments, you can construct the path dynamically like i did. it looks like this

$requirementLine = "/Volumes/catalog_$Env" + "_$Region/packages/latest/suite_data-0.1.0-py3-none-any.whl"
#some code
Add-Content -Path $requirementsPath -Value $requirementLine

Adding the volume as resource is useless in this case because you cannot access the volume when the app is building. It appears that the volume is not mounted yet at that stage. So the app doesn't recognize it. 

PS: It's important to restart your app in the CI/CD pipeline each time there's a new version of you wheel file so it re-installs it again. Otherwise it won't update the libraries with a simple re-deployment.

[nk-five1]

Thank you very much. I hope translate your tips to my case which does not use asset bundles.