requirements.txt with cluster libraries

[sujan1]

Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.

How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of the repository? If absolute, how to specify it?

[462098 · Accepted solution]

To use the new "requirements.txt" feature in your cluster do the following:

1. Change your cluster's "Databricks Runtime Version" to 15.0 or greater (example: "15.4 LTS ML (includes Apache Spark 3.5.0, Scala 2.12)"). 
2. Navigate to the cluster's: "Libraries" tab -> "Install new" button -> "(o) Workspace".
3. Just below the "Workspace File Path" you should now see the text "requirements.txt" added to this prompt: "Choose from workspace or upload a whl/wheelhouse.zip/jar/zip/tar/requirements.txt file to install as a library"
4. Navigate to and select the proper Workspace folder or Repo to select the proper "requirements.txt" file to use for your cluster. NOTE: Any "requirements.txt" file within the Workspace folders structure should now show as bold (not grayed out). 
5. Click the "Install" button. 

[mishravk7250]

how to install requirement.txt using github action.

- name: Install workspace requirements.txt on cluster

env:

CLUSTER_ID: ${{ secrets.DATABRICKS_CLUSTER_ID }}

run: |

databricks libraries install \

--cluster-id "$CLUSTER_ID" \

--whl "dbfs:/FileStore/engineering_workflows/requirements.txt"
But this does not work