cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

requirements.txt with cluster libraries

sujan1
New Contributor II

Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.

How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of the repository? If absolute, how to specify it?

1 ACCEPTED SOLUTION

Accepted Solutions

462098
New Contributor III

To use the new "requirements.txt" feature in your cluster do the following:

  1. Change your cluster's "Databricks Runtime Version" to 15.0 or greater (example: "15.4 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)"). 
  2. Navigate to the cluster's: "Libraries" tab -> "Install new" button -> "(o) Workspace".
  3. Just below the "Workspace File Path" you should now see the text "requirements.txt" added to this prompt: "Choose from workspace or upload a whl/wheelhouse.zip/jar/zip/tar/requirements.txt file to install as a library"
  4. Navigate to and select the proper Workspace folder or Repo to select the proper "requirements.txt" file to use for your cluster. NOTE: Any "requirements.txt" file within the Workspace folders structure should now show as bold (not grayed out). 
  5. Click the "Install" button. 

View solution in original post

2 REPLIES 2

462098
New Contributor III

To use the new "requirements.txt" feature in your cluster do the following:

  1. Change your cluster's "Databricks Runtime Version" to 15.0 or greater (example: "15.4 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)"). 
  2. Navigate to the cluster's: "Libraries" tab -> "Install new" button -> "(o) Workspace".
  3. Just below the "Workspace File Path" you should now see the text "requirements.txt" added to this prompt: "Choose from workspace or upload a whl/wheelhouse.zip/jar/zip/tar/requirements.txt file to install as a library"
  4. Navigate to and select the proper Workspace folder or Repo to select the proper "requirements.txt" file to use for your cluster. NOTE: Any "requirements.txt" file within the Workspace folders structure should now show as bold (not grayed out). 
  5. Click the "Install" button. 

mishravk7250
New Contributor II

how to install requirement.txt using github action.

- name: Install workspace requirements.txt on cluster
env:
CLUSTER_ID: ${{ secrets.DATABRICKS_CLUSTER_ID }}
run: |
databricks libraries install \
--cluster-id "$CLUSTER_ID" \
--whl "dbfs:/FileStore/engineering_workflows/requirements.txt"
But this does not work

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now