I am referencing this doc to initialize a databricks.sdk.WorkspaceClient object instance via Azure Native Authentication. I am initializing this WorkspaceClient within a databricks notebook, but I am trying to use the client to access the Jobs api of a different databricks workspace.
The error I am getting says:
ValueError: default auth: azure-client-secret: Cannot resolve Azure Databricks workspace: b'{"error":{"code":"MissingSubscription","message":"The request did not have a subscription or a valid tenant level resource provider."}}'.
Here is a heavily redacted picture of the code I am running:
I believe my issue might be with the azure_workspace_resource_id field. Currently, I am filling that field with the workspace ID for the databricks workspace for which I would like to run WorkspaceClient.jobs methods. The docs describe this field as:
"The Azure Resource Manager ID for the Azure Databricks workspace, which is exchanged for a Databricks host URL."
Why am I doing this:
I want to check the status of a Databricks job that runs outside of my current Databricks workspace. For production, I would like to authenticate with the Jobs API without using my own personal access token, and the Databricks OAuth M2M docs suggest using the WorkspaceClient to accomplish this.
Questions:
- Is the azure_workspace_resource_id not the same as the Databricks workspace ID?
- Does this Azure service principle need to be added as a Databricks service principle in order for this authentication to succeed?
Thank you.
