run datarbicks worflow as service pricipal (managed identity) reads from azure dev ops repo Failed

[jannemanson]

Hello,

we are running a workflow as a service principal, that is a aad managed identity. This does result in the issue: run databricks workflow as service principal the reads from azure dev ops repo Failed to checkout Git repository: PERMISSION_DENIED: Encountered an error with your Azure Active Directory credentials. Please try logging out of Azure Active Directory (Azure Portal) and logging back in.

If I run the workflow as my own user role it works. The Managed Identity is set as a Contributor in the AzureDevOps repository and has the same rights, as my own user has. The only difference is, that it is a aad manged identity and my role is a aad user. 

[Debayan]

Hi, this issue has to be checked from the AAD end. Could you please check the credentials and confirm?

[IvanK]

@jannemansonDid you manage to solve this issue? If so, how? We are having the same problem

[m997al]

@Debayan I'm having the same issue.  I'm trying to run a Databricks workflow using an Azure service principal (not a Databricks service principal).  The Azure service principal shows up in Databricks (via SCIM provisioning) as a "user" in Databricks.

The documentation from Azure on allowing the Azure service principal to clone a git repo is confusing, especially in light of how the git settings in the task of the Databricks workflow are set up.  

I tried initially using a Databricks service principal, for which I could generate a client_id and a client_secret_id.  However, of course the Databricks service principal has no access at all to Azure DevOps (git repos).  So that didn't work.

It would be great if someone published an example of how to do this.  Azure service principal running Databricks workflows that clone Azure DevOps repos to run code. 

Thanks!

[IvanK]

We managed to solve this problem, however it is not an elegant solution. Databricks should simplify this.

The steps that have to be done are listed below. We are using user assigned managed identity (MI), but I assume this should work for Azure Service Principals as well.

Step 1 - Create a PAT

Create a PAT in Azure DevOps with scope Code (Read). This is done with your own account in Azure DevOps

Step 2 - Generate Databricks Entra ID access token for the MI

Note: This requires you to have attached the MI to a VM or VMSS, to connect to a bash terminal as the MI.

To get the access token, run the following command:

az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d​

Note: that the resource ID 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d corresponds to Azure Databricks programmatic ID [1].

Step 3 - Create Databricks Git credentials for the MI

Use the REST API documentation for creating the Databricks Git credentials [2].

Use the generated access token as "Authorization: Bearer token". For the body use the PAT you created and your Azure DevOps username (your email):

{ "personal_access_token": "<paste-devops-PAT-here>", "git_username": "<paste-your-devops-username>", "git_provider": "azureDevOpsServices" } 

[1] https://learn.microsoft.com/en-us/azure/databricks/dev-tools/service-prin-aad-token#--get-a-microsof...

[2] https://docs.databricks.com/api/azure/workspace/gitcredentials/create