cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

michael_mehrten
New Contributor III

Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:

  1. A user / personal access token
  2. A service principal access token

Using a user access token authenticates the REST API as the user, so all repos actions are performed as the user identity. This isn't desirable for automation, as all automation tasks are tied to a specific user account. In this case, a service principal would be preferable. As far as I can tell, the service principal doesn't work in Azure DevOps, because the service principal doesn't have access to the Azure DevOps git repo.

Has anyone had success getting a service principal access to Azure DevOps? If not, what alternatives have people used to integrate Databricks Repos with Azure DevOps CI/CD (apart from using personal access tokens)?

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

@Michael Mehrtens​, This is now supported. To use a service principal with Repos API first add the Git PAT token for the service principal via the Git Credential API. You can then use Repos API and Jobs APIs with your service principal.

View solution in original post

27 REPLIES 27

michael_mehrten
New Contributor III

My best guess at how we could achieve this is to create a user identity for CI/CD in Azure DevOps, and configure the Service Principal to use that personal access token for Azure DevOps. However, that configuration lives in the "User settings" pane and isn't configurable for Service Principals via the CLI / REST API. Anyone have a good way to modify "User settings" for a service principal?

Anonymous
Not applicable

Hello, @Michael Mehrtens​ . Welcome and thank you for your question! My name is Piper, and I'm a moderator for Databricks. Let's see how the members respond. We'll come back if necessary.

Hey @Piper Wilson​  - any chance we can circle back to this?

Anonymous
Not applicable

Absolutely. I apologize for the delay. I will bump this up to the SMEs.

alexott
Valued Contributor II

Right now it's not possible. There are several reasons - primarily because you can connect to DevOps only using the DevOps personal access token, not the service principal, and there is no REST API to set DevOps PAT programmatically as it's required for service principal. As I know, this API will be added, but not sure about the timeframe yet.

Yann
New Contributor II

Hi,

I have a related question and would like to get a confirmation. We are using a service principal to manage Databricks jobs through Jenkins CI/CD. However, it seems that I can't add a Git integration for the service principal breaking our Jenkins pipeline.

Is it possible or not to add Git integration to a service principal?

Thanks for your time.

Kaniz_Fatma
Community Manager
Community Manager

Hi @Yann ORIEULT​ , Azure doesn't provide the ability to issue a service principle to access git repositories.

Ben_Templeton__
New Contributor III

There is mention of the future ability to use Service Principals with the Repos API here: https://community.databricks.com/s/question/0D53f00001VJn01CAD/repos-configuration-for-azure-service...

Does anyone here know anything about that?

Martin1337
New Contributor II

Any updates on this?​

Anonymous
Not applicable

@Michael Mehrtens​, This is now supported. To use a service principal with Repos API first add the Git PAT token for the service principal via the Git Credential API. You can then use Repos API and Jobs APIs with your service principal.

jrosend
New Contributor III

Any idea on how to accomplish this without using Azure Devops? Our repos are on GitHub and I'm not sure how we can create a GitHub PAT for the service principal in this situation.

I know this is a really old thread, but I still don't understand how this answers the question.

The Git Credential API allows us to create the credentials no problem 👍, but how do we get a Git PAT for a service principal in Azure DevOps? it doesn't seem possible.

  • Service principals can't create tokens, like personal access tokens (PATs) or SSH Keys. They can generate their own Azure AD tokens and these tokens can be used to call Azure DevOps REST APIs.

Source

So as far as I can tell the Azure AD tokens expire after a short duration, so it would require Databricks to hit the OAuth2 endpoint first to get the token, then use that for the git credentials?

I'm hoping I'm just missing something, and there is a way to set this up.

oops, sorry I didn't click the load more replies button and didn't realise there was tons more posts 😂

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group