requirements.txt with cluster libraries

sujan1
New Contributor II

‎05-02-2024 02:02 AM

Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.

How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of the repository? If absolute, how to specify it?

462098
New Contributor III

‎10-24-2024 09:57 AM

To use the new "requirements.txt" feature in your cluster do the following:

  1. Change your cluster's "Databricks Runtime Version" to 15.0 or greater (example: "15.4 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)"). 
  2. Navigate to the cluster's: "Libraries" tab -> "Install new" button -> "(o) Workspace".
  3. Just below the "Workspace File Path" you should now see the text "requirements.txt" added to this prompt: "Choose from workspace or upload a whl/wheelhouse.zip/jar/zip/tar/requirements.txt file to install as a library"
  4. Navigate to and select the proper Workspace folder or Repo to select the proper "requirements.txt" file to use for your cluster. NOTE: Any "requirements.txt" file within the Workspace folders structure should now show as bold (not grayed out). 
  5. Click the "Install" button. 

View solution in original post

mishravk7250
New Contributor II

‎04-23-2025 02:47 AM

how to install requirement.txt using github action.

- name: Install workspace requirements.txt on cluster
env:
CLUSTER_ID: ${{ secrets.DATABRICKS_CLUSTER_ID }}
run: |
databricks libraries install \
--cluster-id "$CLUSTER_ID" \
--whl "dbfs:/FileStore/engineering_workflows/requirements.txt"
But this does not work
0 Kudos