How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

[michael_mehrten]

Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:

1. A user / personal access token
2. A service principal access token

Using a user access token authenticates the REST API as the user, so all repos actions are performed as the user identity. This isn't desirable for automation, as all automation tasks are tied to a specific user account. In this case, a service principal would be preferable. As far as I can tell, the service principal doesn't work in Azure DevOps, because the service principal doesn't have access to the Azure DevOps git repo.

Has anyone had success getting a service principal access to Azure DevOps? If not, what alternatives have people used to integrate Databricks Repos with Azure DevOps CI/CD (apart from using personal access tokens)?

[michael_mehrten]

My best guess at how we could achieve this is to create a user identity for CI/CD in Azure DevOps, and configure the Service Principal to use that personal access token for Azure DevOps. However, that configuration lives in the "User settings" pane and isn't configurable for Service Principals via the CLI / REST API. Anyone have a good way to modify "User settings" for a service principal?

[Kaniz]

Hi @Michael Mehrtens​ , Please have a look - https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-porta...

[Anonymous]

Hello, @Michael Mehrtens​ . Welcome and thank you for your question! My name is Piper, and I'm a moderator for Databricks. Let's see how the members respond. We'll come back if necessary.

[michael_mehrten]

Hey @Piper Wilson​  - any chance we can circle back to this?

[Anonymous]

Absolutely. I apologize for the delay. I will bump this up to the SMEs.

[alexott]

Right now it's not possible. There are several reasons - primarily because you can connect to DevOps only using the DevOps personal access token, not the service principal, and there is no REST API to set DevOps PAT programmatically as it's required for service principal. As I know, this API will be added, but not sure about the timeframe yet.

[Yann]

Hi,

I have a related question and would like to get a confirmation. We are using a service principal to manage Databricks jobs through Jenkins CI/CD. However, it seems that I can't add a Git integration for the service principal breaking our Jenkins pipeline.

Is it possible or not to add Git integration to a service principal?

Thanks for your time.

[Kaniz]

Hi @Yann ORIEULT​ , Azure doesn't provide the ability to issue a service principle to access git repositories.

[Ben_Templeton__]

There is mention of the future ability to use Service Principals with the Repos API here: https://community.databricks.com/s/question/0D53f00001VJn01CAD/repos-configuration-for-azure-service...

Does anyone here know anything about that?

[Martin1337]

Any updates on this?​

[Anonymous · Accepted solution]

@Michael Mehrtens​, This is now supported. To use a service principal with Repos API first add the Git PAT token for the service principal via the Git Credential API. You can then use Repos API and Jobs APIs with your service principal.

[jrosend]

Any idea on how to accomplish this without using Azure Devops? Our repos are on GitHub and I'm not sure how we can create a GitHub PAT for the service principal in this situation.

[terrymunro]

I know this is a really old thread, but I still don't understand how this answers the question.

The Git Credential API allows us to create the credentials no problem 👍, but how do we get a Git PAT for a service principal in Azure DevOps? it doesn't seem possible.

• Service principals can't create tokens, like personal access tokens (PATs) or SSH Keys. They can generate their own Azure AD tokens and these tokens can be used to call Azure DevOps REST APIs.

---

Source

So as far as I can tell the Azure AD tokens expire after a short duration, so it would require Databricks to hit the OAuth2 endpoint first to get the token, then use that for the git credentials?

I'm hoping I'm just missing something, and there is a way to set this up.

[terrymunro]

oops, sorry I didn't click the load more replies button and didn't realise there was tons more posts 😂