topic Error connecting Databricks Notebook using managed identity from Azure Data Factory in Data Engineering

Error connecting Databricks Notebook using managed identity from Azure Data Factory

Prashanth24 — Tue, 06 Aug 2024 09:50:33 GMT

I am trying to connect Databricks Notebook using managed identity authentication type from Azure Data Factory. Below are the settings done. Error message is appended at the bottom of this message. With the same settings but with different authentication type like "Basic", "Service Prinicipal" is working fine. Please let me know anything i am missing out here.

URL => https://adb-xxxxxxxxxxxxxxxxxx.xx.azuredatabricks.net/api/2.0/jobs/runs/submit

Method => POST

Body => {"existing_cluster_id":"xxxx-xxxxxx-xxxxxxxx","notebook_task":{"notebook_path":"/Workspace/Users/<user email id>/workflows/primary_notebook","base_parameters":{"param_1":"First Web Activity"}}}

Authentication => System-assigned Managed Identity

Resources => https://adb-xxxxxxxxxxxxxxxx.xx.azuredatabricks.net/

Error is as below

Get access token from MSI failed for Datafactory data-factory1-us-east-2, region eu2. Please verify resource url is valid and retry. Details: Failed to get MI access token. The error message is: Acquire MI token from AAD failed. ErrorCode: invalid_resource, Message: AADSTS500011: The resource principal named https://adb-xxxxxxxxxxxxxxx.xx.azuredatabricks.net/ was not found in the tenant named xxxxxxxxxxxxxx. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.

Re: Error connecting Databricks Notebook using managed identity from Azure Data Factory

Retired_mod — Thu, 08 Aug 2024 07:43:11 GMT

Hi @Prashanth24, To resolve this, ensure the resource URL is correctly set, grant the Data Factory Managed Identity Contributor role in the Databricks workspace, verify the Databricks workspace is registered in the correct Azure AD tenant, confirm the Managed Identity is properly set up and has necessary permissions, and double-check the Linked Service configuration in Azure Data Factory, making sure the authentication type is "Managed Service Identity" and the workspace resource ID is correct.

If the problem persists, please let me know, and we can explore further troubleshooting steps.

Re: Error connecting Databricks Notebook using managed identity from Azure Data Factory

mujeeb — Wed, 25 Sep 2024 09:16:29 GMT

Hi Kaniz,

I have assigned the managed service identity of adf, the contributor role. I am trying to create a linked service for databricks, but the connection is failing saying "The operation has timed out."

Re: Error connecting Databricks Notebook using managed identity from Azure Data Factory

Hesareal — Mon, 10 Feb 2025 10:08:15 GMT

Did you manage to solve it?
I am getting the same error calling DBX REST API from ADF with System-assigned Managed Identity. The Databricks workspace is added a linked service and there no problem to run notebooks.