topic Reading data from Serverless Warehouse from Azure Functions in Python - using managed identities in Data Engineering

Reading data from Serverless Warehouse from Azure Functions in Python - using managed identities

MRTN — Mon, 02 Mar 2026 15:50:12 GMT

We are trying to run a simple service on an Azure Function app, where we need to query some data from a Databricks Warehouse. We want to avoid managing secrets, and hence try to use Microsoft Entra authentication all the way. Using various available online sources - we have tried the following steps.

Enabled a system assigned "Managed Identity" on the Azure Function app
Created a service principal in the Databricks workspace - "Microsoft Entra ID managed". Linked the service principal to the Azure function by pasting in the application_id
Given the service principal access to the Warehouse (Use)
Give service principal USE access to catalog and schema, SELECT access to relevant table

from databricks import sql from azure.identity import DefaultAzureCredential, AzureCliCredential credential=DefaultAzureCredential() token=credential.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token with sql.connect( server_hostname=DATABRICKS_HOST, http_path=DATABRICKS_WAREHOSE, access_token=token) as connection: with connection.cursor() as cursor: cursor.execute(f"SELECT * FROM {CATALOG}.{SCHEMA}.{TABLE}") result = cursor.fetchall() df=pd.DataFrame(result, columns=[desc[0] for desc in cursor.description])

When testing the Azure Function running locally (effectively using AzureCLICredentials and my own access), this works fine. However, when running the code on the function app itself it returns a 403: Forbidden error.

When creating a OAuth secret in the principal, the table can be read using the following code. I guess this confirms that the service principal has the right access in Databricks. However, we would like to avoid managing secrets.

from databricks.sdk.core import Config, oauth_service_principal def credential_provider(): config = Config( host = f"https://{server_hostname}", client_id = client_id, client_secret = token) return oauth_service_principal(config) with sql.connect(server_hostname = server_hostname, http_path = DATABRICKS_WAREHOSE, credentials_provider = credential_provider) as connection: with connection.cursor() as cursor: cursor.execute(f"SELECT * FROM {CATALOG}.{SCHEMA}.{TABLE}") result = cursor.fetchall() df=pd.DataFrame(result, columns=[desc[0] for desc in cursor.description]) df.head()

Any suggestions on further actions? What are we missing if anything?

Re: Reading data from Serverless Warehouse from Azure Functions in Python - using managed identities

MRTN — Mon, 02 Mar 2026 21:24:33 GMT

We are also trying to follow the instructions given here, creating a .databrickscfg file with

[AZURE_MI_WORKSPACE] host = https://XXX.azuredatabricks.net/ azure_workspace_resource_id = YYY azure_client_id = ZZZ azure_use_msi = true

And removing the token from the function call. This - again - also works locally, while we get the error "Tried all the ports [8020, 8021, 8022, 8023, 8024] for oauth redirect, but can't find free port" when running on an Azure function.

Re: Reading data from Serverless Warehouse from Azure Functions in Python - using managed identities

MRTN — Tue, 03 Mar 2026 14:21:56 GMT

We figured it out. Need create an Azure User Assigned Identity, and add this to the function app. The we added this user managed identity as a service principal in the Databricks workspace The following code then works

credential=ManagedIdentityCredential(client_id=USER_ASSIGNED_IDENTITY_CLIENT_ID) token=credential.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token with sql.connect( server_hostname=DATABRICKS_HOST, http_path=DATABRICKS_WAREHOSE, access_token=token) as connection: with connection.cursor() as cursor: cursor.execute(f"SELECT 'albert' AS name") result = cursor.fetchall() df=pd.DataFrame(result, columns=[desc[0] for desc in cursor.description])

The USER_ASSIGNED_CLIENT_ID is here the Client ID of the user assigned idenity.

Hope to be able to use the System Managed Identity in the future, to avoid the extra steps.

Hi @MRTN, The 403 Forbidden error you are seeing when usi...

SteveOstrowski — Mon, 09 Mar 2026 05:50:05 GMT

Hi @MRTN,

The 403 Forbidden error you are seeing when using DefaultAzureCredential from the Azure Function (while it works locally with AzureCliCredential) comes down to a key distinction in how the token is being used and who the token represents.

UNDERSTANDING THE ISSUE

When you run locally with AzureCliCredential, the token is issued for your personal user identity (which is already added to the Databricks workspace). When the Azure Function runs with its system-assigned managed identity, the token is issued for a different identity, the managed identity itself. Even though you created a service principal in Databricks and linked it by application_id, there is an important nuance: the managed identity and the Microsoft Entra ID managed service principal may not be the same entity in Entra ID unless they share the same application (client) ID.

Here is what to verify and a recommended approach.

STEP 1: CONFIRM THE IDENTITY ALIGNMENT

The system-assigned managed identity of your Azure Function has its own Object ID and a corresponding Application ID in Entra ID. When you created the service principal in Databricks as "Microsoft Entra ID managed," you need to ensure the Application ID you used matches exactly the client ID of the Azure Function's managed identity.

To find your managed identity's client ID:

- Go to the Azure Portal
- Navigate to your Function App
- Click Identity in the left panel
- Under "System assigned," note the Object ID
- You can find the corresponding Application ID in Enterprise Applications in Entra ID by searching for the Object ID

The Application ID of that enterprise application entry is what you must use when adding the service principal to Databricks.

STEP 2: VERIFY THE TOKEN AUDIENCE/SCOPE

In your code, you are requesting a token with this scope:

credential = DefaultAzureCredential()
token = credential.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token

The resource ID 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d is correct for Azure Databricks. That part looks right.

STEP 3: USE THE MICROSOFT ENTRA ID TOKEN AUTH PATTERN CORRECTLY

When you pass the token via access_token to the SQL connector, the connector treats it as a Microsoft Entra ID token (same as a PAT in terms of the parameter). This should work, but the token must represent an identity that is recognized in the Databricks workspace. If the managed identity's Application ID does not match the service principal registered in Databricks, the workspace will reject it with 403.

STEP 4: CONSIDER USING OAUTH M2M AS A CREDENTIALS PROVIDER (RECOMMENDED)

If you want a fully secretless approach that is well-supported by the Databricks SQL Connector, you can combine Azure's DefaultAzureCredential with a custom credentials provider. This avoids passing a static access_token and instead lets the connector handle token refresh through the credentials_provider pattern:

from databricks import sql
from azure.identity import DefaultAzureCredential

DATABRICKS_RESOURCE_ID = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

def credential_provider():
  credential = DefaultAzureCredential()
  def inner():
      token = credential.get_token(f"{DATABRICKS_RESOURCE_ID}/.default")
      return {"Authorization": f"Bearer {token.token}"}
  return inner

with sql.connect(
  server_hostname=DATABRICKS_HOST,
  http_path=DATABRICKS_WAREHOUSE,
  credentials_provider=credential_provider
) as connection:
  with connection.cursor() as cursor:
      cursor.execute(f"SELECT * FROM {CATALOG}.{SCHEMA}.{TABLE}")
      result = cursor.fetchall()

This approach uses the credentials_provider parameter instead of access_token, which gives the connector the ability to refresh the token automatically. It also lets DefaultAzureCredential resolve the correct credential source: AzureCliCredential locally, and ManagedIdentityCredential when deployed to the Azure Function.

STEP 5: ALTERNATIVE, EXPLICIT MANAGED IDENTITY CREDENTIAL

If you want to be fully explicit about using the managed identity (and avoid fallback behavior of DefaultAzureCredential), you can use ManagedIdentityCredential directly:

from databricks import sql
from azure.identity import ManagedIdentityCredential

DATABRICKS_RESOURCE_ID = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

# For system-assigned managed identity, no client_id needed
credential = ManagedIdentityCredential()

# For user-assigned managed identity, pass the client_id:
# credential = ManagedIdentityCredential(client_id="<your-managed-identity-client-id>")

def credential_provider():
  def inner():
      token = credential.get_token(f"{DATABRICKS_RESOURCE_ID}/.default")
      return {"Authorization": f"Bearer {token.token}"}
  return inner

with sql.connect(
  server_hostname=DATABRICKS_HOST,
  http_path=DATABRICKS_WAREHOUSE,
  credentials_provider=credential_provider
) as connection:
  with connection.cursor() as cursor:
      cursor.execute(f"SELECT * FROM {CATALOG}.{SCHEMA}.{TABLE}")
      result = cursor.fetchall()

DEBUGGING TIPS

1. Decode your token to inspect it. You can use jwt.ms or the following Python code to see what identity the token represents:

import jwt
decoded = jwt.decode(token, algorithms=["RS256"], options={"verify_signature": False})
print(decoded.get("appid"))  # This should match the SP registered in Databricks
print(decoded.get("aud"))    # Should be 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d

2. Confirm the appid from the decoded token matches the Application ID of the service principal you added in Databricks.

3. Double-check that the service principal has CAN USE permission on the SQL warehouse, plus USE CATALOG, USE SCHEMA, and SELECT on the target table.

RELEVANT DOCUMENTATION

- Databricks SQL Connector authentication: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/python-sql-connector
- Microsoft Entra ID token auth: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/aad-token-manual
- Azure managed identities with Databricks: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/azure-mi-auth
- Managed identities for Azure resources overview: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.