Private Python Package in Serverless Job

[vinitkhandelwal]

I am trying to create a Databricks Job using Serverless Compute. I am using wheel file to run the Python Job.

The wheel file has setup.py file using which all dependencies are installed. One of the package dependency is a private package hosted on Gitlab Python Package Registry.  Now this package installation needs some access details to be provided to the compute. Earlier when I was not using Serverless compute, I was able to add init_script to compute and that would run before setup.py, thereby giving access to private registry to install the private package.

The init_script added registry login details to /etc/pip.conf file as follows:

#!/bin/bash
if [[ $GITLAB_TOKEN ]]; then
use $GITLAB_TOKEN 
fi
echo $PYPI_TOKEN
printf "[global]\n" > /etc/pip.conf
printf "extra-index-url =\n" >> /etc/pip.conf
printf "\thttps://__token__:$GITLAB_TOKEN@gitlab.com/api/v4/projects/12345678/packages/pypi/simple\n" >> /etc/pip.conf

 But now with Serverless, there is no option to add init scripts, so how do I allow the compute to be able to install private package?

[vinitkhandelwal]

@Retired_mod  Can you guide, how can I achieve your goal of installing a private package hosted on Gitlab Python Package Registry?

[djhs]

With a pre-signed URL you mean that I need to expose my Gitlab token in a the index URL? Is there a way inject the Gitlab token without exposing it? I haven't found one thus far.