how to read reuqiremnet.txt in databrick workspace

[CE]

Dear databrick team,

I want to know if there is a method in Databricks equivalent to pip install -r requirements.txt

There are packages I want to install in this path: /Workspace/Users/xxx@domain.com/databrick_requirement.txt

I have referred to the following document: https://learn.microsoft.com/en-us/azure/databricks/libraries/notebooks-python-libraries#use-a-requir...

I executed:
%pip install -r /Workspace/Users/xxx@domain.com/databrick_requirement.txt
but it gave the following error:

Looking forward to your reply, thank you!

[filipniziol · Accepted solution]

Hi @CE ,

Yes, you can install packages from a requirements.txt file in Databricks, similar to using pip install -r requirements.txt in a local environment. However, the path you provided (/Workspace/Users/xxx@domain.com/databrick_requirement.txt) isn't directly accessible by the %pip command.

1. If you have a unity catalog available, create a Volume, upload your file to the volume and then run your pip install command like:

%pip install -r "/Volumes/<path to the file>/requirements.txt"

2. If you using the old workspace without unity catalog, do similarly, but upload the file to DBFS

3. Also, instead of running %pip install within individual notebooks, a more efficient approach—especially for managing dependencies across multiple notebooks and users—is to use init scripts. This ensures that all necessary libraries are installed automatically when the cluster starts.

[Panda]

@CE 

You can't directly access /Workspace paths like a traditional filesystem path. When you specify /Workspace/Users/xxx@domain.com/databrick_requirement.txt, %pip install cannot interpret it because the %pip magic command works with DBFS paths. Follow below approach

Approach 1:- 

Copy your databrick_requirement.txt file to DBFS, which %pip install can access directly:

%pip install -r /dbfs/databrick_requirement.txt

Approach 2:

Instead of directly installing dependencies from databrick_requirement.txt, you can create wheel files (.whl) for your Python packages. This approach can improve consistency, as wheel files are precompiled and can be installed much faster.

1. Generate Wheels: pip wheel -r databrick_requirement.txt -w ./wheels
2. Either directly upload via the UI or use the Databricks CLI to copy the wheel files to a DBFS location.
3. Install the Wheels in Databricks: Install the packages from the .whl files:%pip install /dbfs/wheels/my_package-1.0.0-py3-none-any.whl

Benefits of Using Wheels:

Faster installation: Wheels are precompiled, so they install faster compared to source distributions.

Consistency: Precompiled wheels ensure consistent versions across different environments.