Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Showing results for 
Search instead for 
Did you mean: 

URGENT: dbt Job Failing in Databricks - Azure Repo Access Denied (Service Principal)

New Contributor

I am encountering issues while running a Databricks job using a Microsoft Entra ID Service Principal. My workflow includes a task of type "dbt," which requires authentication and access to the Azure Repo containing my dbt project code. I have granted admin-level permissions to this Service Principal in the Azure Databricks workspace, SQL Warehouse Compute, etc. I also added this Service Principal to the Azure DevOps project and granted it Read and Contributor permissions on the dbt Repo. Additionally, I elevated permissions by assigning this Service Principal as a Project Administrator in DevOps.

Despite these settings, when I run the workflow, it reports that it doesn't have permissions to check-out the repo, with the error message:

"run failed with error message
Failed to check-out Git repository: PERMISSION_DENIED: Encountered an error with your Azure Active Directory credentials. Please try logging out of Azure Active Directory ( and logging back in."

I have reviewed various documentation but have not found clear guidance on how to run a job using a Service Principal that requires authentication and access to Azure Repos. I have also logged a ticket with Microsoft Azure Databricks support, but have not yet received a solution. The support engineer mentioned they are consulting with the Product team.

I would greatly appreciate any guidance on this issue. Thanks in advance.


Community Manager
Community Manager

Hi @SumitBhatia

  • You’ll need to create a service principal in your Microsoft Entra ID (formerly Azure Active Directory) tenant. This service principal will represent your application and allow it to authenticate with Azure services.
  • Make sure you have the necessary permissions to register an application in your Microsoft Entra ID tenant.
  • A PAT is required to authenticate to the Databricks REST API. You can create one in your Azure Databricks workspace.
  • Use this token to authenticate your requests when interacting with Databricks.
  • Use the Databricks SCIM API to add the service principal as a non-administrative user to your Azure Databricks workspace.
  • This step ensures that the service principal has the necessary permissions within Databricks.
  • In Azure Databricks, create a secret scope backed by Azure Key Vault.
  • This secret scope will allow you to securely store secrets (such as credentials) needed by your job.
  • Grant the service principal read access to the secret scope.
  • This ensures that your job can access the necessary secrets (e.g., database credentials) securely.
  • Configure the job cluster to read secrets from the secret scope you created.
  • Transfer ownership of the job to the service principal.
  • Remember that you cannot use a cluster with credential passthrough enabled to run a job owned by a service principal. If your job requires a service principal to access Azure storage, refer to the documentation on conn...

I hope this guidance helps you resolve the issue. If you have any further questions or need additional assistance, feel free to ask! 😊

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!