Error connecting Databricks Notebook using managed identity from Azure Data Factory

Prashanth24
New Contributor III

I am trying to connect Databricks Notebook using managed identity authentication type from Azure Data Factory. Below are the settings done. Error message is appended at the bottom of this message. With the same settings but with different authentication type like "Basic", "Service Prinicipal" is working fine. Please let me know anything i am missing out here.

URL => https://adb-xxxxxxxxxxxxxxxxxx.xx.azuredatabricks.net/api/2.0/jobs/runs/submit

Method => POST

Body => {"existing_cluster_id":"xxxx-xxxxxx-xxxxxxxx","notebook_task":{"notebook_path":"/Workspace/Users/<user email id>/workflows/primary_notebook","base_parameters":{"param_1":"First Web Activity"}}}

Authentication => System-assigned Managed Identity

Resources => https://adb-xxxxxxxxxxxxxxxx.xx.azuredatabricks.net/

Error is as below

Get access token from MSI failed for Datafactory data-factory1-us-east-2, region eu2. Please verify resource url is valid and retry. Details: Failed to get MI access token. The error message is: Acquire MI token from AAD failed. ErrorCode: invalid_resource, Message: AADSTS500011: The resource principal named https://adb-xxxxxxxxxxxxxxx.xx.azuredatabricks.net/ was not found in the tenant named xxxxxxxxxxxxxx. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.