requirements.txt with cluster libraries

[sujan1]

Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.

How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of the repository? If absolute, how to specify it?

[Kaniz]

To specify the requirements.txt file path for libraries in a job cluster workflow in Databricks, you have a few options.

Let’s break it down:

1. Upload the requirements.txt File:

   • First, upload your requirements.txt file to your Databricks workspace.
   • You can do this by navigating to the workspace, selecting the appropriate folder, and uploading the file.

2. Specify the Workspace File Path:

   • When creating or configuring a job cluster, you can specify the location of the requirements.txt file using the “Workspace File Path” field.
   • Enter the workspace location of your requirements.txt file in the following format:

     /Workspace/Users/someone@example.com/<path-to-library>/<library-name>.<ext>
     

     Replace <path-to-library> with the actual path to your requirements.txt file within the workspace, a...¹.

3. Relative vs. Absolute Paths:

   • The workspace file path you provide can be either relative or absolute.
   • If you use a relative path, it will be relative to the root of your Databricks workspace.
   • If you use an absolute path, it should start with /Workspace/....

Remember to adjust the path according to your specific workspace structure and the location of your requirements.txt file. Once configured, your job cluster will use the specified requirements.txt for library installation d...².

Additionally, if you’re working with Databricks Asset Bundles, dependencies for local development are specified in the requirements*.txt file at the root of the bundle project. However, job task library dependencies are declared in your bundle configuration files and are often...³.

I hope this helps! Let me know if you have any further questions. 😊