Service Principal types in Azure Databricks

[darkolexis]

In Azure Databricks, we can create two types of Service Principals, namely:

1. Databricks Managed SP

2. Microsoft Entra ID Managed SP

What is the difference between two, other than one being specific to single workspace, and another being usable from multiple Databricks workspaces? Can someone define the detailed properties of both?

[szymon_dybczak]

Hi @darkolexis ,

So the main difference is that in case of Azure Databricks managed service principals you  can authenticate to Azure Databricks using Databricks OAuth authentication and personal access tokens.

In case of Microsoft Entra ID managed service principals you can authenticate to Azure Databricks using Databricks OAuth authentication and Microsoft Entra ID tokens. 

Another difference is Azure Databricks managed service principals are managed directly within Azure Databricks. Microsoft Entra ID managed service principals are managed in Microsoft Entra ID, which requires additional permissions.

Databricks recommends that you use Azure Databricks managed service principals for Azure Databricks automation and that you use Microsoft Entra ID managed service principals in cases where you must authenticate with Azure Databricks and other Azure resources at the same time.

So for example, a good use case for Microsoft Entra Id managed service principal is situation when you'd like to

get access to some azure services, for example configuring databricks auto loader with file notification mode requires EntraID Service Principal with following roles assigned:

• Contributor: This role is for setting up resources in your storage account, such as queues and event subscriptions.
• Storage Queue Data Contributor: This role is for performing queue operations such as retrieving and deleting messages from the queues. This role is required only when you provide a service principal without a connection string.

In such case you can't use it Databricks managed service principa

[arunprakash1986]

So, what use would it be in a situation where I have a Docker image that runs as a job using Databricks Compute. Here the Job has "Run As" which is set to a service principal, say "svc1" which is a databricks managed service principal. I believe that this job when trying to access unity catalog tables will apply restrictions based on the permissions of "svc1". 

If there was another "svcazure1" which is a Azure Managed identity, and if I use it as the "Run As", apart from the unity catalog restrictions, what else can this be used for, from the code that's in the Docker. Can this identity access Azure Key Vault if it has access to it? If yes, do you know how this can be achieved in Python?