topic Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps? in Data Engineering

How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

michael_mehrten — Tue, 02 Nov 2021 17:05:39 GMT

Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:

A user / personal access token
A service principal access token

Using a user access token authenticates the REST API as the user, so all repos actions are performed as the user identity. This isn't desirable for automation, as all automation tasks are tied to a specific user account. In this case, a service principal would be preferable. As far as I can tell, the service principal doesn't work in Azure DevOps, because the service principal doesn't have access to the Azure DevOps git repo.

Has anyone had success getting a service principal access to Azure DevOps? If not, what alternatives have people used to integrate Databricks Repos with Azure DevOps CI/CD (apart from using personal access tokens)?

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

michael_mehrten — Tue, 02 Nov 2021 17:20:08 GMT

My best guess at how we could achieve this is to create a user identity for CI/CD in Azure DevOps, and configure the Service Principal to use that personal access token for Azure DevOps. However, that configuration lives in the "User settings" pane and isn't configurable for Service Principals via the CLI / REST API. Anyone have a good way to modify "User settings" for a service principal?

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Anonymous — Tue, 02 Nov 2021 19:26:10 GMT

Hello, @Michael Mehrtens . Welcome and thank you for your question! My name is Piper, and I'm a moderator for Databricks. Let's see how the members respond. We'll come back if necessary.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

michael_mehrten — Thu, 11 Nov 2021 18:20:27 GMT

Hey @Piper Wilson - any chance we can circle back to this?

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Anonymous — Thu, 11 Nov 2021 18:53:30 GMT

Absolutely. I apologize for the delay. I will bump this up to the SMEs.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

alexott — Thu, 25 Nov 2021 18:43:10 GMT

Right now it's not possible. There are several reasons - primarily because you can connect to DevOps only using the DevOps personal access token, not the service principal, and there is no REST API to set DevOps PAT programmatically as it's required for service principal. As I know, this API will be added, but not sure about the timeframe yet.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Yann — Wed, 05 Jan 2022 11:37:43 GMT

Hi,

I have a related question and would like to get a confirmation. We are using a service principal to manage Databricks jobs through Jenkins CI/CD. However, it seems that I can't add a Git integration for the service principal breaking our Jenkins pipeline.

Is it possible or not to add Git integration to a service principal?

Thanks for your time.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Ben_Templeton__ — Fri, 08 Apr 2022 05:09:11 GMT

There is mention of the future ability to use Service Principals with the Repos API here: https://community.databricks.com/s/question/0D53f00001VJn01CAD/repos-configuration-for-azure-service-principal

Does anyone here know anything about that?

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Martin1337 — Mon, 09 May 2022 20:53:24 GMT

Any updates on this?

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Anonymous — Mon, 13 Jun 2022 15:31:44 GMT

@Michael Mehrtens, This is now supported. To use a service principal with Repos API first add the Git PAT token for the service principal via the Git Credential API. You can then use Repos API and Jobs APIs with your service principal.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

jrosend — Tue, 29 Nov 2022 20:06:57 GMT

Any idea on how to accomplish this without using Azure Devops? Our repos are on GitHub and I'm not sure how we can create a GitHub PAT for the service principal in this situation.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

xiangzhu — Tue, 10 Jan 2023 21:52:54 GMT

Hello,

My use case is to create dbt task inside of a databricks workflow.

It needs to specify `git_source`, and my workflow is run under a service principal account.

Unfortunately, from the beginning of the workflow run. an error is raised like:

"Failed to checkout Git repository: PERMISSION_DENIED: Encountered an error with your Azure Active Directory credentials. Please try logging out of Azure Active Directory (https://portal.azure.com) and logging back in."

But I've no where to grant to the service principal the Azure DevOps checkout permission.

Replacing the service principal with an standard user account works, but we can not use user account in production.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

Anonymous — Tue, 10 Jan 2023 22:41:23 GMT

Are you able to use Azure DevOps Personal Access token for this instead of Active Directory credentials? Using DevOps PATS directly should work with service principals.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

xiangzhu — Tue, 10 Jan 2023 23:03:45 GMT

yes, I tried adding PAT in the git_source, it doesn't work neither.

I'm sure the PAT and the URL is OK, as I can `git clone (git_source url value with PAT inside)` locally without any issue.

tested with runtime: 10.4.x-cpu-ml-scala2.12

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

171499 — Thu, 23 Feb 2023 14:55:44 GMT

@Vaibhav Sethi "first add the Git PAT token for the service principal via the Git Credential API. " - is this possible? On the Git Credentials API guide I only see the option to set the credentials for the current user (not a service principal)

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

xiangzhu — Thu, 23 Feb 2023 22:38:56 GMT

yes, it is. you could find a quick tuto from this link:

Repos configuration for Azure Service Principal (databricks.com)

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

terrymunro — Tue, 25 Jul 2023 05:51:28 GMT

I know this is a really old thread, but I still don't understand how this answers the question.

The Git Credential API allows us to create the credentials no problem 👍, but how do we get a Git PAT for a service principal in Azure DevOps? it doesn't seem possible.

Service principals can't create tokens, like personal access tokens (PATs) or SSH Keys. They can generate their own Azure AD tokens and these tokens can be used to call Azure DevOps REST APIs.

Source

So as far as I can tell the Azure AD tokens expire after a short duration, so it would require Databricks to hit the OAuth2 endpoint first to get the token, then use that for the git credentials?

I'm hoping I'm just missing something, and there is a way to set this up.

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

terrymunro — Tue, 25 Jul 2023 05:52:45 GMT

oops, sorry I didn't click the load more replies button and didn't realise there was tons more posts 😂

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

rsenjins — Mon, 31 Jul 2023 10:28:33 GMT

How did you solve this? Where did you find a way to create PAT tokens for Service Principals? The other comments don't make it that clear either for me

Re: How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?

terrymunro — Mon, 31 Jul 2023 12:03:30 GMT

Unfortunately I didn't find any solution to this. 🙁