cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Repos configuration for Azure Service Principal

pantelis_mare
Contributor III

Hello community!

I would like to update a repo from within my Azure DevOps release pipeline.

In the pipeline I generate a token using a AAD Service Principal as recommended, and I setup the databricks api using that token.

When I pass the databricks repos update command, I receive an authenitcation error, which is expected and the service principal has not git configured on the workspace side.

My question is:

Can I configure the repos for the SPN programmatically?

Or, is there a way to provide an Azure Devops token when I make the databricks api call? I have tried passing a token by setting the git AZURE_DEVOPS_EXT_PAT but it doesn't seem to work.

Thank you in advance!

31 REPLIES 31

davidfengler
New Contributor II

Microsoft released "Service principal and managed identity support in Azure DevOps

" end of march -> https://learn.microsoft.com/en-us/azure/devops/release-notes/2023/sprint-219-update#service-principa...

does this helps to get this working? my job shows the same issue as before when executing with the service principal without setting git_credentials

xiangzhu
Contributor III

traditional PAT may have long lifespn, but the new SP feature uses an AAD token which should have a much shorter lifespqn, maybe around one hour, this could be a limiting factor.

However, I haven't tested this yet, so these are merely hypotheses.

Nevertheless, even if it does work with SP, running dbt or notebooks through Git or workspace does not seem like a good idea in terms of operational risk. Packaging everything into an artifact, such as a spark_python_task, appears to be a much better approach.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group