cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

run datarbicks worflow as service pricipal (managed identity) reads from azure dev ops repo Failed

jannemanson
New Contributor III

Hello,

we are running a workflow as a service principal, that is a aad managed identity. This does result in the issue: run databricks workflow as service principal the reads from azure dev ops repo Failed to checkout Git repository: PERMISSION_DENIED: Encountered an error with your Azure Active Directory credentials. Please try logging out of Azure Active Directory (Azure Portal) and logging back in.

If I run the workflow as my own user role it works. The Managed Identity is set as a Contributor in the AzureDevOps repository and has the same rights, as my own user has. The only difference is, that it is a aad manged identity and my role is a aad user. 

4 REPLIES 4

Debayan
Databricks Employee
Databricks Employee

Hi, this issue has to be checked from the AAD end. Could you please check the credentials and confirm?

IvanK
New Contributor III

@jannemansonDid you manage to solve this issue? If so, how? We are having the same problem

m997al
Contributor III

@Debayan I'm having the same issue.  I'm trying to run a Databricks workflow using an Azure service principal (not a Databricks service principal).  The Azure service principal shows up in Databricks (via SCIM provisioning) as a "user" in Databricks.

The documentation from Azure on allowing the Azure service principal to clone a git repo is confusing, especially in light of how the git settings in the task of the Databricks workflow are set up.  

I tried initially using a Databricks service principal, for which I could generate a client_id and a client_secret_id.  However, of course the Databricks service principal has no access at all to Azure DevOps (git repos).  So that didn't work.

It would be great if someone published an example of how to do this.  Azure service principal running Databricks workflows that clone Azure DevOps repos to run code. 

Thanks!

IvanK
New Contributor III

We managed to solve this problem, however it is not an elegant solution. Databricks should simplify this.

The steps that have to be done are listed below. We are using user assigned managed identity (MI), but I assume this should work for Azure Service Principals as well.

Step 1 - Create a PAT

Create a PAT in Azure DevOps with scope Code (Read). This is done with your own account in Azure DevOps

Step 2 - Generate Databricks Entra ID access token for the MI

Note: This requires you to have attached the MI to a VM or VMSS, to connect to a bash terminal as the MI.

To get the access token, run the following command:

 
az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d​
 
Note: that the resource ID 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d corresponds to Azure Databricks programmatic ID [1].

Step 3 - Create Databricks Git credentials for the MI

Use the REST API documentation for creating the Databricks Git credentials [2].

Use the generated access token as "Authorization: Bearer token". For the body use the PAT you created and your Azure DevOps username (your email):

{ "personal_access_token": "<paste-devops-PAT-here>", "git_username": "<paste-your-devops-username>", "git_provider": "azureDevOpsServices" } 

[1] https://learn.microsoft.com/en-us/azure/databricks/dev-tools/service-prin-aad-token#--get-a-microsof...

[2] https://docs.databricks.com/api/azure/workspace/gitcredentials/create

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group