Error authenticating databricks.sdk.WorkspaceClient with external workspace via Azure Native Auth

[hayden_blair]

I am referencing this doc to initialize a databricks.sdk.WorkspaceClient object instance via Azure Native Authentication. I am initializing this WorkspaceClient within a databricks notebook, but I am trying to use the client to access the Jobs api of a different databricks workspace. 

The error I am getting says:

ValueError: default auth: azure-client-secret: Cannot resolve Azure Databricks workspace: b'{"error":{"code":"MissingSubscription","message":"The request did not have a subscription or a valid tenant level resource provider."}}'.

Here is a heavily redacted picture of the code I am running:

I believe my issue might be with the azure_workspace_resource_id field. Currently, I am filling that field with the workspace ID for the databricks workspace for which I would like to run WorkspaceClient.jobs methods. The docs describe this field as:

"The Azure Resource Manager ID for the Azure Databricks workspace, which is exchanged for a Databricks host URL."

Why am I doing this:

I want to check the status of a Databricks job that runs outside of my current Databricks workspace. For production, I would like to authenticate with the Jobs API without using my own personal access token, and the Databricks OAuth M2M docs suggest using the WorkspaceClient to accomplish this.

Questions:

1. Is the azure_workspace_resource_id not the same as the Databricks workspace ID?
2. Does this Azure service principle need to be added as a Databricks service principle in order for this authentication to succeed?

Thank you.

[Kaniz_Fatma]

Hi @hayden_blair, It seems you’re encountering an issue with Azure Native Authentication while trying to access the Jobs API of a different Databricks workspace using the databricks.sdk.WorkspaceClient.

Let’s break down your questions:

1. Is the azure_workspace_resource_id the same as the Databricks workspace ID?

   • The azure_workspace_resource_id is not the same as the Databricks workspace ID. It represents the Azure Resource Manager ID for the Azure Databricks workspace. This ID is exchanged for a Databricks host URL during authentication. Essentially, it acts as a bridge between Azure and Databricks to establish the connection.
   • When using the WorkspaceClient, you should provide the correct azure_workspace_resource_id associated with the target Databricks workspace.

2. Does the Azure service principal need to be added as a Databricks service principal for successful authentication?

   • Yes, for successful authentication, the Azure service principal (used for Azure Native Authentication) needs to be added as a Databricks service principal.
   • You can provision a service principal specifically for Azure Databricks automation. Once created, add this service principal to either the Azure Databricks account, a specific workspace, or both.
   • By doing so, you’ll be able to authenticate at the Azure Databricks account level or with a specific workspace, allowing you to access the Jobs API without relying on your personal access token.

Remember to verify that the service principal has the necessary permissions (such as Contributor or Owner) on the Azure workspace resource. This ensures a smooth authentication process. If you encounter any further issues, feel free to ask! 😊