Install python packages from Azure DevOps feed with service principal authentication

[Marco37]

At the moment I install python packages from our Azure DevOps feed with a PAT token as authentication mechanism. This works well, but I want to use a service principal instead of the PAT token.

• I have created an Azure service principal and assigned it the "Feed Reader" permission on the Azure DevOps feed.
• I have added it's secret to an Azure keyvault and databricks has permission to read this secret
• In databricks I have created a secret scope pointing to this keyvault
• Within my cluster policy I have defined these environment variables:
  AZ_DEVOPS_PROJECT_NAME=<project>
  AZ_DEVOPS_TOKEN={{secrets/<keyvault>/<secret name>}}
  AZ_DEVOPS_ORG_NAME=<organization name>
  AZ_DEVOPS_FEED_NAME=<feed name>
  and it points to an init script that contains this script:
  pip config set global.index-url https://${AZ_DEVOPS_FEED_NAME}:${AZ_DEVOPS_TOKEN}@pkgs.dev.azure.com/${AZ_DEVOPS_ORG_NAME}/${AZ_DEVOPS_PROJECT_NAME}/_packaging/${AZ_DEVOPS_FEED_NAME}/pypi/simple/
  
  

The cluster is able to read the secret from the keyvault

The index URL is set to the correct feed

But I'm not able to install a package from the Azure DevOps feed

For the PAT token setup I'm using exactly the same configuration and that works fine.

Is what I'm trying to do not possible, or am I missing something?

Regards,

Marco

[szymon_dybczak]

Hi @Marco37 ,

Did you follow some guideline or documentation when you were trying to configure it?

At first glance it looks incorrect.  In following line you're trying to use service principal as a token? If so it definitely won't work. 

pip config set global.index-url https://${AZ_DEVOPS_FEED_NAME}:${AZ_DEVOPS_TOKEN}@pkgs.dev.azure.com/${AZ_DEVOPS_ORG_NAME}/${AZ_DEVOPS_PROJECT_NAME}/_packaging/${AZ_DEVOPS_FEED_NAME}/pypi/simple name>}}

You can't simply substitute a service principal secret for a PAT token in the URL because they use different authentication flows.

Service principals require OAuth2 flow to obtain access token.

[Marco37]

Hi szymon_dybczak

I have searched a lot for documentation, but is mostly ends with people falling back to the PAT tokens or it just ends with nothing or they are about connecting from Azure DevOps to Databricks (the opposite direction).

Do you have some documentation about using OAuth2, so that I can accomplish my goal?

Kind Regards,

Marco

[szymon_dybczak]

Hi @Marco37 ,

So you need to use client-credential flow to obtain access token. Once you get token, that should work as the password with an arbitrary username, in the same way as PAT would.

Here's how you can obtain access token for service principal using API call:

- OAuth 2.0 client credentials flow on the Microsoft identity platform - Microsoft identity platform |...

Alternatively, you can use artifacts-keyring approach. According to below threads they added support Managed Identity and Service Principal (but I think for SP they only support authentication with certificate)

Is it possible to use artifacts-keyring with a Service Principal? · Issue #60 · microsoft/artifacts-...

Managed Identity and Service Principal Support by embetten · Pull Request #492 · microsoft/artifacts...

Connect your Python project to an Azure Artifacts feed - Azure Artifacts | Microsoft Learn

[Marco37]

Thank a lot szymon_dybczak, 

I did try artifacts-keyring last week, but without success. I'm going to try the API call this coming week and will let you know the results

Kind Regards,

Marco

[Marco37]

I am able to retrieve the access token of my service principal (eyJ0eXAiOiJK....etc.), but I do not know how to use it in my pip config statement. I have tried to replace the value of the variable ${AZ_DEVOPS_TOKEN}, but that doesn't work.

Kind Regards,
Marco

[szymon_dybczak]

Hi @Marco37 ,

According to github thread you can use that access token in following way (just replace your values). Let me know if that works. If you still encounter issues I can try to recreate this scenario in my environment:

TOKEN= your_token
pip install <package> --index-url "https://$TOKEN@<Organization>.pkgs.visualstudio.com/<Project>/_packaging/<Feed>/pypi/simple/"

[Marco37]

Hi szymon_dybczak,

When I use this index URL I get SSL errors (maybe a firewall or whitelist issue?)

[szymon_dybczak · Accepted solution]

Hi @Marco37 ,

Yep, probably this is firewall issue. I've recreated your scenario and I can confirm that approach I've described above will work. So, I created Service Principal and I assigned him Feed Reader permisson. Then I obtained access token using client credential flow and I used that token in following way.

TOKEN = YOUR_TOKEN
ORGANIZATION = YOUR_ORGANIZATION
PROJECT = YOUR_PROJECT

%pip install azure.identity --index-url "https://{TOKEN}@{ORGANIZATION}.pkgs.visualstudio.com/{PROJECT}/_packaging/MyTestFeed/pypi/simple/"

[szymon_dybczak]

I've just noticed that you could also make mistake when constructing your url. You're using ${AZ_DEVOPS_ORG_NAME}, but this is not an f-sting. Try the same approach as mine.

TOKEN = YOUR_TOKEN
ORGANIZATION = YOUR_OGRANIZATION
PROJECT = YOUR_PROJECT

%pip install azure.identity --index-url "https://{TOKEN}@{ORGANIZATION}.pkgs.visualstudio.com/{PROJECT}/_packaging/MyTestFeed/pypi/simple/"

[Marco37]

Hi szymon_dybczak,

I have also tried it in my personal Azure subscription, so that I'm sure that all traffic is allowed, but I still can't get it to work

TOKEN = "eyJ0eXAiOiJKV1QiLCJub25jZSI6InVnSVJ0a1pyLWdi................."
ORGANIZATION = "mhofman37"
PROJECT = "6121f0ba-31c2-407b-80e0-ccd1a57ce8e4"

%pip install adal --index-url "https://{TOKEN}@{ORGANIZATION}.pkgs.visualstudio.com/{PROJECT}/_packaging/test/pypi/simple/"

The service principal has permissions on the feed

Kind Regards,
Marco

[szymon_dybczak]

Hi @Marco37 ,

When you tried again, did you regenerated access token? They are short-lived, so you need to acquire new one. It should work, I've verified it on my environment and I had no issue with authentication using service principal

[Marco37]

Hi szymon_dybczak,

Yes, I have generated a new token each time, because they are only valid for one hour. Tomorrow I will discuss it with my colleagues. I must be overlooking something, because it does work in your environment.

I will let you know when I figure it out.

Thanks a lot for your help,
Marco