cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

requirements.txt with cluster libraries

sujan1
New Contributor

Cluster libraries are supported from version 15.0 - Databricks Runtime 15.0 | Databricks on AWS.

How can I specify requirements.txt file path in the libraries in a job cluster in my workflow? Can I use relative path? Is it relative from the root of the repository? If absolute, how to specify it?

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

To specify the requirements.txt file path for libraries in a job cluster workflow in Databricks, you have a few options.

Let’s break it down:

  1. Upload the requirements.txt File:

    • First, upload your requirements.txt file to your Databricks workspace.
    • You can do this by navigating to the workspace, selecting the appropriate folder, and uploading the file.
  2. Specify the Workspace File Path:

  3. Relative vs. Absolute Paths:

    • The workspace file path you provide can be either relative or absolute.
    • If you use a relative path, it will be relative to the root of your Databricks workspace.
    • If you use an absolute path, it should start with /Workspace/....

Remember to adjust the path according to your specific workspace structure and the location of your requirements.txt file. Once configured, your job cluster will use the specified requirements.txt for library installation d...2.

Additionally, if you’re working with Databricks Asset Bundles, dependencies for local development are specified in the requirements*.txt file at the root of the bundle project. However, job task library dependencies are declared in your bundle configuration files and are often...3.

I hope this helps! Let me know if you have any further questions. 😊

 
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!