cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Install bundle built artifact as notebook-scoped library

keb
New Contributor II

We are having a hard time finding an intuitive way of using the artifacts we build and deploy with databricks bundle deploy notebook-scoped.

Desired result:
Having internal artifacts be available notebook-scoped for jobs by config
or
Having an easier way of %pip installing internal artifacts

Our setup:
Monorepo of multiple bundles and a package of module of shared functionality

This snippet in all bundles:

artifacts:
  common-lib:
    type: "whl"
    path: "../common-lib"

This ensures that the common-lib is built and included in artifacts/.internal/ for each bundle deployment
Then we specify this for every task that needs it:

libraries:
  - whl: ${workspace.artifact_path}/.internal/common-0.1-py3-none-any.whl

This works, but we found out that this installs the library at the cluster level.
This makes many entries in the libraries tabs for the clusters pointing to internal artifacts to the bundles.

Working "solution":
We have gotten it to work by doing this:

# For each job.yml under resources
parameters:
  - name: common-lib-path
    default: ${workspace.artifact_path}/.internal/common-0.1-py3-none-any.whl
# On top of notebooks that needs the common lib

# Cell 1
lib_path = dbutils.widgets.get("common-lib-path")

# Cell 2
%pip install $lib_path

It works, but it is not a proper solution.

Is there a good way of achieving this we have not found?



1 REPLY 1

keb
New Contributor II

We were not able to find a clean solution for this, so what we ended up doing is referencing the common lib like this in every notebook it is needed.

%pip install ../../../artifacts/.internal/common-0.1-py3-none-any.whl

 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now