10-23-2024 03:01 AM
Hi,
We have issues trying to run Databricks notebooks orchestrated with Azure Data Factory. We have been doing this for a while now without any issues when we use Job Clusters, Existing General Purpose Clusters, or Cluster Pools. We use an Azure Data Factory Managed Service Identity (service principal) that we have integrated into our Databricks workspace.
The problem is when we try to use an existing Serverless SQL Warehouse. We are able to get the ID and all necessary parameters. When we test the connection it is successful. However, we are not able to run the notebook. We get the error:
"Run aborted because the job run-as lacks Attach permissions on the underlying cluster"
As shown below.
However, I am able to run the same notebook successfully when I use my PAT to connect to the Serverles warehouse.
Any idea on how to solve the issue? We really don't want to run our jobs based on personal credentials...
10-25-2024 12:18 AM
No one? 😕
10-25-2024 07:39 AM
Hi @ArturOA ,
Maybe you forget to give permission to ADF MSI to this serverless warehouse? Check how's your permission tab looks like.
11-05-2024 03:58 AM
Hei @szymon_dybczak ,
Your suggestion only allows giving permissions to individuals. We need to give permission to a Service Principal, and this is not possible.
It seems it is not allowed by design, unfortunately...
11-05-2024 09:39 AM
Hi @ArturOA ,
I think you're wrong here. Let's have a look at below screenshot. I'm able to add permission to ADF managed identity to Serveless Warehouse. You can also create group and put service principal/managed identity inside this group and give permission to entire group.
11-05-2024 05:52 AM
@ArturOA you can try adding a service principal in AD group and Add that AD Group to the server permissions.
11-06-2024 02:51 AM
@ArturOA Have you synced this Managed Identity of ADF as SPN to Databricks?
a month ago
Does the service principal has access and permission for the notebook?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group