topic Re: Repos configuration for Azure Service Principal in Data Engineering

Repos configuration for Azure Service Principal

pantelis_mare — Mon, 22 Nov 2021 16:24:40 GMT

Hello community!

I would like to update a repo from within my Azure DevOps release pipeline.

In the pipeline I generate a token using a AAD Service Principal as recommended, and I setup the databricks api using that token.

When I pass the databricks repos update command, I receive an authenitcation error, which is expected and the service principal has not git configured on the workspace side.

My question is:

Can I configure the repos for the SPN programmatically?

Or, is there a way to provide an Azure Devops token when I make the databricks api call? I have tried passing a token by setting the git AZURE_DEVOPS_EXT_PAT but it doesn't seem to work.

Thank you in advance!

Re: Repos configuration for Azure Service Principal

jose_gonzalez — Mon, 29 Nov 2021 19:12:44 GMT

Hi @Pantelis Maroudis ,

Let me reach out to the SME group to get help on your question.

Re: Repos configuration for Azure Service Principal

Srihasa_Akepati — Tue, 30 Nov 2021 07:43:34 GMT

@Pantelis Maroudis I would like to comment on the Azure SP supportability feature - Updating a databricks repo using Repos API to authenticate using Azure Service Principal AAD is not supported yet. Active work is in progress, We can expect it by Jan 2022.

Re: Repos configuration for Azure Service Principal

pantelis_mare — Thu, 09 Dec 2021 08:13:11 GMT

Thank you @Srihasa Reddy Akepati . Looking forward to it 🙂

Re: Repos configuration for Azure Service Principal

JakeP — Mon, 14 Mar 2022 16:13:18 GMT

Hello, curious if any update on this? I too would like to use AAD token auth (as service principal) to call repos POST and PATCH from an Azure DevOps pipeline. Thanks!

Re: Repos configuration for Azure Service Principal

pantelis_mare — Mon, 14 Mar 2022 16:28:34 GMT

Hello @Jake Panchyshyn

From my side I used a separate token generated from account for all steps in my pipeline that needed to use the repos api

Re: Repos configuration for Azure Service Principal

JakeP — Mon, 14 Mar 2022 16:44:57 GMT

Thanks, though, could you elaborate as to what kind of token you generate as what account or identity? Are you authenticating as a user (rather than service principal) or ??

Re: Repos configuration for Azure Service Principal

Kirk1 — Thu, 24 Mar 2022 08:52:31 GMT

Is this supported now given that it is past the Jan 2022 date stated above? Because I have found that a SPN still can't authenticate via AAD.

Re: Repos configuration for Azure Service Principal

venkad — Sat, 26 Mar 2022 02:12:23 GMT

Looking forward for this update

Re: Repos configuration for Azure Service Principal

Ben_Templeton__ — Fri, 08 Apr 2022 05:06:36 GMT

@Srihasa Reddy Akepati Any update on this?

Re: Repos configuration for Azure Service Principal

Srihasa_Akepati — Fri, 08 Apr 2022 08:01:25 GMT

@Ben Templeton Iam checking internally on the status of this feature. I will update the thread once i receive response.

Re: Repos configuration for Azure Service Principal

Kirk1 — Fri, 08 Apr 2022 08:04:03 GMT

I got this response from Databricks on this topic: "The AAD support depends on Microsoft implementing token exchange for ADO. This is underway but we don't have a timeline for it."

Re: Repos configuration for Azure Service Principal

pantelis_mare — Fri, 08 Apr 2022 08:31:44 GMT

Sorry for the late reply @Jake Panchyshyn .

Yes, I use a token generated from my account to interact with the repos API in my pipeline. For the rest of the operations (eg job creation) I use the token generated by the service principal

Re: Repos configuration for Azure Service Principal

Martin1337 — Mon, 09 May 2022 20:53:01 GMT

Has there been any updates here? Or any viable workaround?

Re: Repos configuration for Azure Service Principal

FabriceDeseyn — Tue, 10 May 2022 12:10:48 GMT

The solution depends on accepting ServicePrincipal tokens credentials as authentication for ADO. There is mention of it on their roadmap (https://docs.microsoft.com/en-us/azure/devops/release-notes/features-timeline) but no timeline is defined yet.

As a workaround we use a user PAT for the time being.

Re: Repos configuration for Azure Service Principal

Kirk1 — Fri, 22 Jul 2022 12:10:41 GMT

I see that microsoft has a draft timeline regarding this..... FY23Q1 (TBC): Public Preview to all internal and external customers. I don't know why it takes them so long.

Re: Repos configuration for Azure Service Principal

AlexMethod — Thu, 01 Dec 2022 21:39:30 GMT

Any updates?

Re: Repos configuration for Azure Service Principal

Dave_B_ — Fri, 02 Dec 2022 01:23:46 GMT

I was able to accomplish this by getting an AAD service account (not the service principal used by CI/CD) and got rights to use that account to connect to our github repo. I used the databricks git config api to configure the service principal user git config with the AAD service account and PAT that I generated and authorized in github. I then used the Repos CLI to create the repo in databricks under the service principal. I just got this working today. Note that I'm still trying to figure out how to update from github as we are not making any changes within databricks itself.

Re: Repos configuration for Azure Service Principal

xiangzhu — Tue, 10 Jan 2023 21:51:59 GMT

My use case is to create dbt task inside of a databricks workflow.

It needs to specify `git_source`, and my workflow is run under a service principal account.

Unfortunately, from the beginning of the workflow run. an error is raised like:

"Failed to checkout Git repository: PERMISSION_DENIED: Encountered an error with your Azure Active Directory credentials. Please try logging out of Azure Active Directory (https://portal.azure.com) and logging back in."

But I've no where to grant to the service principal the Azure DevOps checkout permission.

Replacing the service principal with an standard user account works, but we can not use user account in production.

Re: Repos configuration for Azure Service Principal

gentresh — Thu, 16 Feb 2023 13:42:11 GMT

Dave, do you have a moment to guide me through these steps of yours?

What do you mean by, "got rights to use that account to connect to our github repo". What steps would you need to follow, or are you referring to an approval?
The reason I ask is that, we recently changed from a user-based to a service-based approach, and while the Azure Service Principal has not changed, I get a PERMISSION_DENIED - cannot read repo.
I think that because of our recent change, I will need to assign the Github PAT (new) token to the Service Principal, however I see no way of doing that using the Databricks GUI Workspace.
Can you point me to any documentation for: "I used the databricks git config api to configure the service principal user git config with the AAD service account and PAT that I generated and authorized in github"
PS: I'm a databricks newb that has inherited a rather complex Azure landing zone.

Help would be more than appreciated!!!