I am encountering issues while running a Databricks job using a Microsoft Entra ID Service Principal. My workflow includes a task of type "dbt," which requires authentication and access to the Azure Repo containing my dbt project code. I have granted admin-level permissions to this Service Principal in the Azure Databricks workspace, SQL Warehouse Compute, etc. I also added this Service Principal to the Azure DevOps project and granted it Read and Contributor permissions on the dbt Repo. Additionally, I elevated permissions by assigning this Service Principal as a Project Administrator in DevOps.
Despite these settings, when I run the workflow, it reports that it doesn't have permissions to check-out the repo, with the error message:
"run failed with error message
Failed to check-out Git repository: PERMISSION_DENIED: Encountered an error with your Azure Active Directory credentials. Please try logging out of Azure Active Directory (https://portal.azure.com) and logging back in."
I have reviewed various documentation but have not found clear guidance on how to run a job using a Service Principal that requires authentication and access to Azure Repos. I have also logged a ticket with Microsoft Azure Databricks support, but have not yet received a solution. The support engineer mentioned they are consulting with the Product team.
I would greatly appreciate any guidance on this issue. Thanks in advance.
