Private Python Package in Serverless Job

[777433]

I am trying to create a Databricks Job using Serverless Compute. I am using wheel file to run the Python Job.

The wheel file has setup.py file using which all dependencies are installed. One of the package dependency is a private package hosted on Gitlab Python Package Registry.  Now this package installation needs some access details to be provided to the compute. Earlier when I was not using Serverless compute, I was able to add init_script to compute and that would run before setup.py, thereby giving access to private registry to install the private package.

The init_script added registry login details to /etc/pip.conf file as follows:

#!/bin/bash
if [[ $GITLAB_TOKEN ]]; then
use $GITLAB_TOKEN 
fi
echo $PYPI_TOKEN
printf "[global]\n" > /etc/pip.conf
printf "extra-index-url =\n" >> /etc/pip.conf
printf "\thttps://__token__:$GITLAB_TOKEN@gitlab.com/api/v4/projects/12345678/packages/pypi/simple\n" >> /etc/pip.conf

 But now with Serverless, there is no option to add init scripts, so how do I allow the compute to be able to install private package?

[Kaniz_Fatma]

Hi @777433, When using Serverless Compute in Databricks, you’re right that there’s no direct option to add init scripts. However, you can still achieve your goal of installing a private package hosted on Gitlab Python Package Registry.