Error authenticating databricks.sdk.WorkspaceClient with external workspace via Azure Native Auth

hayden_blair — Mon, 29 Jan 2024 21:03:16 GMT

I am referencing this doc to initialize a databricks.sdk.WorkspaceClient object instance via Azure Native Authentication. I am initializing this WorkspaceClient within a databricks notebook, but I am trying to use the client to access the Jobs api of a different databricks workspace.

The error I am getting says:

ValueError: default auth: azure-client-secret: Cannot resolve Azure Databricks workspace: b'{"error":{"code":"MissingSubscription","message":"The request did not have a subscription or a valid tenant level resource provider."}}'.

Here is a heavily redacted picture of the code I am running:

I believe my issue might be with the azure_workspace_resource_id field. Currently, I am filling that field with the workspace ID for the databricks workspace for which I would like to run WorkspaceClient.jobs methods. The docs describe this field as:

"The Azure Resource Manager ID for the Azure Databricks workspace, which is exchanged for a Databricks host URL."

Why am I doing this:

I want to check the status of a Databricks job that runs outside of my current Databricks workspace. For production, I would like to authenticate with the Jobs API without using my own personal access token, and the Databricks OAuth M2M docs suggest using the WorkspaceClient to accomplish this.

Questions:

Is the azure_workspace_resource_id not the same as the Databricks workspace ID?
Does this Azure service principle need to be added as a Databricks service principle in order for this authentication to succeed?

Thank you.

topic Error authenticating databricks.sdk.WorkspaceClient with external workspace via Azure Native Auth in Get Started Discussions

Error authenticating databricks.sdk.WorkspaceClient with external workspace via Azure Native Auth