Databricks Community

SKiruthiga · Monday

Hi everyone,

I appreciate your time!! We currently use username and password authentication to connect to Snowflake from a Databricks Spark job. We are exploring Workload Identity Federation as an alternative so that we can eliminate long-lived Snowflake credentials.

However, it appears that the Snowflake Spark connector may not currently support Azure Workload Identity Federation for Spark DataFrame read/write operations.

Has anyone successfully configured this authentication flow from Azure Databricks to Snowflake? If so, could you please share the supported connector version, required Databricks configuration, and Snowflake-side setup?

Thank you for your help.

Best regards,
Kiruthiga

Louis_Frolio · Monday

Hello @SKiruthiga, I took a look at both internal and external documentation and here is what I found.

Short version: the Snowflake Connector for Spark does support workload identity federation, starting with version 3.1.7 (certified against JDBC 3.28.0). The harder question is whether your Azure Databricks cluster can hand the connector a usable Azure identity. Those are two different things.

The connector options look like this, and the same options work for df.write:

sfOptions = {
    "sfURL": "<account_identifier>.snowflakecomputing.com",
    "sfUser": "<snowflake_service_user>",
    "sfAuthenticator": "WORKLOAD_IDENTITY",
    "sfWorkloadIdentityProvider": "AZURE",
    "sfDatabase": "<database>",
    "sfSchema": "<schema>",
    "sfWarehouse": "<warehouse>"
}

df = spark.read.format("snowflake").options(**sfOptions).option("dbtable", "<table>").load()

On the Snowflake side, create a service user whose identity matches the Azure identity the job runs as, then grant it only the role and privileges it needs:

CREATE USER <snowflake_service_user>
  TYPE = SERVICE
  WORKLOAD_IDENTITY = (
    TYPE = AZURE
    ISSUER = 'https://login.microsoftonline.com/<tenant_id>/v2.0'
    SUBJECT = '<managed_identity_object_id>'
  );

Now a caveat. AZURE mode expects to fetch a token from the Azure Instance Metadata Service for a managed identity attached to the VM. On Databricks classic compute those VMs sit in the Databricks-managed resource group, so you can't attach your own identity to them, and serverless has no VM you control at all. I don't see anything public that guarantees this endpoint is available to the connector, and I'd expect it to fail. If it does, tweaking the two Spark options won't fix it.

The route I'd try instead is OIDC mode with a Unity Catalog service credential supplying the token. A service credential wraps a managed identity (through an Access Connector for Azure Databricks) and gives your code a TokenCredential. Rough shape:

Tenant admin, one time: consent to Snowflake's multi-tenant Entra app (link in the Snowflake WIF doc). That app is the audience Snowflake expects on Azure tokens.
Azure: create an Access Connector for Azure Databricks and note its managed identity's Object ID.
Databricks: create a UC service credential on that access connector and grant ACCESS to the job's principal.
In the job, request a token and pass it through:

cred = dbutils.credentials.getServiceCredentialsProvider("snowflake-wif")
token = cred.get_token("api://fd3f753b-eed3-462c-b6a7-a4b5bb650aad/.default").token

sfOptions["sfWorkloadIdentityProvider"] = "OIDC"
sfOptions["sfToken"] = token

Snowflake: decode one token first (the WIF doc has a jq one-liner) and copy the exact iss and sub claims into ISSUER and SUBJECT. Managed identity tokens sometimes carry the v1 issuer (sts.windows.net/...) rather than v2, and Snowflake matches exactly. If you use TYPE = OIDC on the user instead of TYPE = AZURE, set OIDC_AUDIENCE_LIST to the token's audience too.

I haven't run the OIDC plus service credential combination end to end, so treat it as the approach I'd try rather than a confirmed recipe. And check which connector version your Databricks Runtime bundles (ls /databricks/jars | grep -i snowflake on the cluster). If it's older than 3.1.7 you'll need to attach the newer connector and JDBC jars and confirm they win over the bundled ones.

A few smaller notes. Entra tokens last about an hour; that's fine for a batch job that fetches a fresh one per run, but a long-running streaming job needs a refresh plan. Don't confuse Snowflake WIF with Databricks OAuth token federation, which is the reverse direction (external workloads authenticating into Databricks) and a separate trust setup. And whichever path you pick, validate with a SELECT 1 read and a tiny write before migrating the real job.

If WIF is a dead end in your setup, Snowflake External OAuth with a short-lived Entra token (sfAuthenticator = oauth) is the fallback. It needs a Snowflake security integration and its own token refresh handling, but it still gets you off stored passwords. If your need is mostly reads, Lakehouse Federation's Snowflake connection with OAuth keeps auth out of your job code entirely.

References:

Snowflake WIF overview, including Azure setup and supported drivers: https://docs.snowflake.com/en/user-guide/workload-identity-federation
Snowflake Spark connector options: https://docs.snowflake.com/en/user-guide/spark-connector-use
Databricks: read and write Snowflake data: https://learn.microsoft.com/en-us/azure/databricks/connect/external-systems/snowflake
Create UC service credentials on Azure: https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-services/service-cred...
Use service credentials from code: https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-services/use-service-...

If you get it working, please post back with the ISSUER and SUBJECT shape you ended up with. It'd save the next person a lot of trial and error.

Regards, Louis.

SKiruthiga · Monday

Thank you Louis, surely will try it out and update you!!

Kiruthiga

balajij8 · Monday

@SKiruthiga You can use Lakehouse Federation using Microsoft Entra ID in Machine to Machine mode if feasible. You can shift the authentication part to Unity Catalog. To configure this flow, you can register an application in Entra ID. On the Snowflake side, you can create an External OAuth security integration tied to that Entra app. You finally create a Unity Catalog foreign connection mapping to Snowflake.

Databricks handles the token exchange automatically in the background. It technically replaces the native Snowflake credentials with an Entra ID client secret and eliminates long lived Snowflake tokens for Spark jobs and manual credential management. Routing it through Lakehouse Federation gives you centralized, governed and SQL based access to Snowflake data directly inside Unity Catalog. More details here

data_pulse · Tuesday

@SKiruthiga

I’m using Snowflake through a Unity Catalog federated connection with PEM private-key authentication. That keeps credentials out of the Spark code and lets Snowflake exposed through a foreign catalog.

It definitely is not the same as Snowflake workload identity federation, but if the main requirement is secure read/query access from Databricks without username/password in the job, it may be worth considering. Lakehouse Federation also supports predicate pushdown to snowflake which helped us pull only the data required.

One limitation is that Federated Connection is primarily for querying/read access, so it won’t replace a Spark connector workflow if you also need to write back to Snowflake. Docs for reference.