I added init script to compute in order to add details of private package login in /etc/pip.conf

Something as follows:

resource "databricks_workspace_file" "gitlab_pypi_init_script" {
  provider = databricks.workspace
  content_base64 = base64encode(<<-EOT
    #!/bin/bash
    if [[ $PYPI_TOKEN ]]; then
    use $PYPI_TOKEN
    fi
    echo $PYPI_TOKEN
    printf "[global]\n" > /etc/pip.conf
    printf "extra-index-url =\n" >> /etc/pip.conf
    printf "\thttps://__token__:$PYPI_TOKEN@gitlab.com/api/v4/projects/12345678/packages/pypi/simple\n" >> /etc/pip.conf
    EOT
  )
  path = "/FileStore/gitlab_pypi_init_script.sh"
}

 I added this file to Workspace and then referenced it under init scripts in cluster compute and it worked to install private package in the cluster when it starts

I also made sure gitlab token was accessible with variable PYPI_TOKEN using Spark environment variables