databricks serverless cluster and poetry private repository
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2025 04:40 AM - edited 10-22-2025 04:41 AM
Currently we are evaluating the databricks serverless. It support public repository in poetry as dependency path but it is not supporting private repository as we are not sure whether put the credentials details regarding privare repository.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2025 05:15 AM
Hi @mahfooz_iiitian ,
Databricks supports private repositories only for Notebook-scoped libraries.
In serverless you can use do it using pip install (of course store you token in a safe palce):
Notebook-scoped Python libraries - Azure Databricks | Microsoft Learn
%pip install --index-url https://<user>:$token@<your-package-repository>.com/<path/to/repo> <package>==<version> --extra-index-url https://pypi.org/simple/Another option for serverless is to create a base environment YAML file. Then just add the --index-url to a base environment YAML file (just like they suggest on screenshot below):
Configure the serverless environment - Azure Databricks | Microsoft Learn
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2025 06:49 AM
I am able to achieve it using pip magic command in notebook as well as package/pip requirements file in base ennvironment. I am specially looking support of poetry with private repository.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-22-2025 07:12 AM - edited 10-22-2025 07:18 AM
To be honest maybe I don't understand your question entirely. Poetry supports private package sources that implement the simple API repository - PEP 503.
"Poetry can fetch and install package dependencies from public or private custom repositories that implement the simple repository API as described in PEP 503."
So, general method of authentication to such repositories is given in above answer - embed credentials directly in the index URL:
pip install --index-url https://<username>:<token>@<repository-host>/<path> <package>
Or if you want you can configure credentials directly in poetry you can do it in following way: