topic how to read reuqiremnet.txt in databrick workspace in Data Engineering

how to read reuqiremnet.txt in databrick workspace

CE — Mon, 14 Oct 2024 03:49:56 GMT

Dear databrick team,

I want to know if there is a method in Databricks equivalent to pip install -r requirements.txt

There are packages I want to install in this path: /Workspace/Users/xxx@domain.com/databrick_requirement.txt

I have referred to the following document: https://learn.microsoft.com/en-us/azure/databricks/libraries/notebooks-python-libraries#use-a-requirements-file-to-install-libraries

I executed:
%pip install -r /Workspace/Users/xxx@domain.com/databrick_requirement.txt
but it gave the following error:

Looking forward to your reply, thank you!

Re: how to read reuqiremnet.txt in databrick workspace

filipniziol — Mon, 14 Oct 2024 06:14:56 GMT

Hi @CE ,

Yes, you can install packages from a requirements.txt file in Databricks, similar to using pip install -r requirements.txt in a local environment. However, the path you provided (/Workspace/Users/xxx@domain.com/databrick_requirement.txt) isn't directly accessible by the %pip command.

1. If you have a unity catalog available, create a Volume, upload your file to the volume and then run your pip install command like:

%pip install -r "/Volumes/<path to the file>/requirements.txt"

2. If you using the old workspace without unity catalog, do similarly, but upload the file to DBFS

3. Also, instead of running %pip install within individual notebooks, a more efficient approach—especially for managing dependencies across multiple notebooks and users—is to use init scripts. This ensures that all necessary libraries are installed automatically when the cluster starts.

Re: how to read reuqiremnet.txt in databrick workspace

Panda — Mon, 14 Oct 2024 12:12:11 GMT

@CE

You can't directly access /Workspace paths like a traditional filesystem path. When you specify /Workspace/Users/xxx@domain.com/databrick_requirement.txt, %pip install cannot interpret it because the %pip magic command works with DBFS paths. Follow below approach

Approach 1:-

Copy your databrick_requirement.txt file to DBFS, which %pip install can access directly:

%pip install -r /dbfs/databrick_requirement.txt

Approach 2:

Instead of directly installing dependencies from databrick_requirement.txt, you can create wheel files (.whl) for your Python packages. This approach can improve consistency, as wheel files are precompiled and can be installed much faster.

Generate Wheels: pip wheel -r databrick_requirement.txt -w ./wheels
Either directly upload via the UI or use the Databricks CLI to copy the wheel files to a DBFS location.
Install the Wheels in Databricks: Install the packages from the .whl files:%pip install /dbfs/wheels/my_package-1.0.0-py3-none-any.whl

Benefits of Using Wheels:

Faster installation: Wheels are precompiled, so they install faster compared to source distributions.

Consistency: Precompiled wheels ensure consistent versions across different environments.