databricks serverless cluster and poetry private repository

[mahfooz_iiitian]

Currently we are evaluating the databricks serverless. It support public repository in poetry as dependency path but it is not supporting private repository as we are not sure whether put the credentials details regarding privare repository.

[szymon_dybczak]

Hi @mahfooz_iiitian ,

Databricks supports private repositories only for Notebook-scoped libraries.

In serverless you can use do it using pip install (of course store you token in a safe palce):

Notebook-scoped Python libraries - Azure Databricks | Microsoft Learn

%pip install --index-url https://<user>:$token@<your-package-repository>.com/<path/to/repo> <package>==<version> --extra-index-url https://pypi.org/simple/

 Another option for serverless is to create a base environment YAML file. Then just add the --index-url to a base environment YAML file (just like they suggest on screenshot below):

Configure the serverless environment - Azure Databricks | Microsoft Learn

[mahfooz_iiitian]

I am able to achieve it using pip magic command in notebook as well as package/pip requirements file in base ennvironment. I am specially looking support of poetry with private repository.

[szymon_dybczak]

To be honest  maybe I don't understand your question entirely. Poetry supports private package sources that implement the simple API repository - PEP 503.

"Poetry can fetch and install package dependencies from public or private custom repositories that implement the simple repository API as described in PEP 503."

So, general method of authentication to such repositories is given in above answer - embed credentials directly in the index URL:

pip install --index-url https://<username>:<token>@<repository-host>/<path> <package>

Or if you want you can configure credentials directly in poetry you can do it in following way: