topic Re: pip install in cluster using web UI and extra index in Data Engineering

pip install in cluster using web UI and extra index

yopbibo — Mon, 17 Apr 2023 14:01:19 GMT

In an init script or a notebook, we can:

pip install --index-url=<our private pypi url> --extra-index-url=https://pypi.org/simple <a module>

In the cluster web UI (libraries -> install library), we can give only the url of our private repository, but no extra-index. That means that if our module gets dependencies to other modules available in pypi, it will fail, fetching them.

Is there a solution, using the web UI?

Re: pip install in cluster using web UI and extra index

Anonymous — Tue, 18 Apr 2023 07:36:49 GMT

Hi @Philippe CRAVE

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: pip install in cluster using web UI and extra index

Debayan — Tue, 18 Apr 2023 07:04:48 GMT

Hi, All the libraries and the dependencies can be packed in the same bundle and uploaded in Web UI, is that feasible?

Found a good read on the same: https://www.activestate.com/resources/quick-reads/how-to-package-python-dependencies-with-pip-setuptools/

Please let us know if this helps.

Also, please tag @Debayan with your next response which will notify me. Thank you!