how to read reuqiremnet.txt in databrick workspace

CE · ‎10-13-2024

Dear databrick team,

I want to know if there is a method in Databricks equivalent to pip install -r requirements.txt

There are packages I want to install in this path: /Workspace/Users/xxx@domain.com/databrick_requirement.txt

I have referred to the following document: https://learn.microsoft.com/en-us/azure/databricks/libraries/notebooks-python-libraries#use-a-requir...

I executed:
%pip install -r /Workspace/Users/xxx@domain.com/databrick_requirement.txt
but it gave the following error:

Looking forward to your reply, thank you!

filipniziol · ‎10-13-2024

Hi @CE ,

Yes, you can install packages from a requirements.txt file in Databricks, similar to using pip install -r requirements.txt in a local environment. However, the path you provided (/Workspace/Users/xxx@domain.com/databrick_requirement.txt) isn't directly accessible by the %pip command.

1. If you have a unity catalog available, create a Volume, upload your file to the volume and then run your pip install command like:

%pip install -r "/Volumes/<path to the file>/requirements.txt"

2. If you using the old workspace without unity catalog, do similarly, but upload the file to DBFS

3. Also, instead of running %pip install within individual notebooks, a more efficient approach—especially for managing dependencies across multiple notebooks and users—is to use init scripts. This ensures that all necessary libraries are installed automatically when the cluster starts.

View solution in original post

Panda · ‎10-14-2024

@CE

You can't directly access /Workspace paths like a traditional filesystem path. When you specify /Workspace/Users/xxx@domain.com/databrick_requirement.txt, %pip install cannot interpret it because the %pip magic command works with DBFS paths. Follow below approach

Approach 1:-

Copy your databrick_requirement.txt file to DBFS, which %pip install can access directly:

%pip install -r /dbfs/databrick_requirement.txt

Approach 2:

Instead of directly installing dependencies from databrick_requirement.txt, you can create wheel files (.whl) for your Python packages. This approach can improve consistency, as wheel files are precompiled and can be installed much faster.

Generate Wheels: pip wheel -r databrick_requirement.txt -w ./wheels
Either directly upload via the UI or use the Databricks CLI to copy the wheel files to a DBFS location.
Install the Wheels in Databricks: Install the packages from the .whl files:%pip install /dbfs/wheels/my_package-1.0.0-py3-none-any.whl

Benefits of Using Wheels:

Faster installation: Wheels are precompiled, so they install faster compared to source distributions.

Consistency: Precompiled wheels ensure consistent versions across different environments.