URGENT: dbt Job Failing in Databricks - Azure Repo Access Denied (Service Principal)

[SumitBhatia]

I am encountering issues while running a Databricks job using a Microsoft Entra ID Service Principal. My workflow includes a task of type "dbt," which requires authentication and access to the Azure Repo containing my dbt project code. I have granted admin-level permissions to this Service Principal in the Azure Databricks workspace, SQL Warehouse Compute, etc. I also added this Service Principal to the Azure DevOps project and granted it Read and Contributor permissions on the dbt Repo. Additionally, I elevated permissions by assigning this Service Principal as a Project Administrator in DevOps.

Despite these settings, when I run the workflow, it reports that it doesn't have permissions to check-out the repo, with the error message:

"run failed with error message
Failed to check-out Git repository: PERMISSION_DENIED: Encountered an error with your Azure Active Directory credentials. Please try logging out of Azure Active Directory (https://portal.azure.com) and logging back in."

I have reviewed various documentation but have not found clear guidance on how to run a job using a Service Principal that requires authentication and access to Azure Repos. I have also logged a ticket with Microsoft Azure Databricks support, but have not yet received a solution. The support engineer mentioned they are consulting with the Product team.

I would greatly appreciate any guidance on this issue. Thanks in advance.

[Kaniz]

Hi @SumitBhatia, 

• You’ll need to create a service principal in your Microsoft Entra ID (formerly Azure Active Directory) tenant. This service principal will represent your application and allow it to authenticate with Azure services.
• Make sure you have the necessary permissions to register an application in your Microsoft Entra ID tenant.
• A PAT is required to authenticate to the Databricks REST API. You can create one in your Azure Databricks workspace.
• Use this token to authenticate your requests when interacting with Databricks.
• Use the Databricks SCIM API to add the service principal as a non-administrative user to your Azure Databricks workspace.
• This step ensures that the service principal has the necessary permissions within Databricks.
• In Azure Databricks, create a secret scope backed by Azure Key Vault.
• This secret scope will allow you to securely store secrets (such as credentials) needed by your job.
• Grant the service principal read access to the secret scope.
• This ensures that your job can access the necessary secrets (e.g., database credentials) securely.
• Configure the job cluster to read secrets from the secret scope you created.
• Transfer ownership of the job to the service principal.
• Remember that you cannot use a cluster with credential passthrough enabled to run a job owned by a service principal. If your job requires a service principal to access Azure storage, refer to the documentation on conn...

I hope this guidance helps you resolve the issue. If you have any further questions or need additional assistance, feel free to ask! 😊