murtazahzaveri
New Contributor II

For Authentication you can provide below config on cluster's Spark Environment Variables,
PIP_EXTRA_INDEX_URL=https://username:password@pkgs.sample.com/sample/_packaging/artifactory_name/pypi/simple/.

Also, you can store the value in Databricks secret