Private Python Package in Serverless Job

vinitkhandelwal — Wed, 03 Jul 2024 15:34:39 GMT

I am trying to create a Databricks Job using Serverless Compute. I am using wheel file to run the Python Job.

The wheel file has setup.py file using which all dependencies are installed. One of the package dependency is a private package hosted on Gitlab Python Package Registry. Now this package installation needs some access details to be provided to the compute. Earlier when I was not using Serverless compute, I was able to add init_script to compute and that would run before setup.py, thereby giving access to private registry to install the private package.

The init_script added registry login details to /etc/pip.conf file as follows:

#!/bin/bash if [[ $GITLAB_TOKEN ]]; then use $GITLAB_TOKEN fi echo $PYPI_TOKEN printf "[global]\n" > /etc/pip.conf printf "extra-index-url =\n" >> /etc/pip.conf printf "\thttps://__token__:$GITLAB_TOKEN@gitlab.com/api/v4/projects/12345678/packages/pypi/simple\n" >> /etc/pip.conf

But now with Serverless, there is no option to add init scripts, so how do I allow the compute to be able to install private package?

Re: Private Python Package in Serverless Job

vinitkhandelwal — Mon, 08 Jul 2024 16:58:58 GMT

@Retired_mod Can you guide, how can I achieve your goal of installing a private package hosted on Gitlab Python Package Registry?

Re: Private Python Package in Serverless Job

djhs — Tue, 08 Oct 2024 11:53:00 GMT

With a pre-signed URL you mean that I need to expose my Gitlab token in a the index URL? Is there a way inject the Gitlab token without exposing it? I haven't found one thus far.

topic Re: Private Python Package in Serverless Job in Get Started Discussions

Private Python Package in Serverless Job

Re: Private Python Package in Serverless Job

Re: Private Python Package in Serverless Job