topic Attaching to Serverless from Azure Data Factory via Service Principal in Data Engineering

Attaching to Serverless from Azure Data Factory via Service Principal

ArturOA — Wed, 23 Oct 2024 10:01:52 GMT

Hi,

We have issues trying to run Databricks notebooks orchestrated with Azure Data Factory. We have been doing this for a while now without any issues when we use Job Clusters, Existing General Purpose Clusters, or Cluster Pools. We use an Azure Data Factory Managed Service Identity (service principal) that we have integrated into our Databricks workspace.

The problem is when we try to use an existing Serverless SQL Warehouse. We are able to get the ID and all necessary parameters. When we test the connection it is successful. However, we are not able to run the notebook. We get the error:

"Run aborted because the job run-as lacks Attach permissions on the underlying cluster"

As shown below.

However, I am able to run the same notebook successfully when I use my PAT to connect to the Serverles warehouse.

Any idea on how to solve the issue? We really don't want to run our jobs based on personal credentials...

Re: Attaching to Serverless from Azure Data Factory via Service Principal

ArturOA — Fri, 25 Oct 2024 07:18:59 GMT

No one? 😕

Re: Attaching to Serverless from Azure Data Factory via Service Principal

szymon_dybczak — Fri, 25 Oct 2024 14:39:18 GMT

Hi @ArturOA ,

Maybe you forget to give permission to ADF MSI to this serverless warehouse? Check how's your permission tab looks like.

Re: Attaching to Serverless from Azure Data Factory via Service Principal

ArturOA — Tue, 05 Nov 2024 11:58:08 GMT

Hei @szymon_dybczak ,

Your suggestion only allows giving permissions to individuals. We need to give permission to a Service Principal, and this is not possible.

It seems it is not allowed by design, unfortunately...

Re: Attaching to Serverless from Azure Data Factory via Service Principal

chvamsi07 — Tue, 05 Nov 2024 13:52:48 GMT

@ArturOA you can try adding a service principal in AD group and Add that AD Group to the server permissions.

Re: Attaching to Serverless from Azure Data Factory via Service Principal

szymon_dybczak — Tue, 05 Nov 2024 17:39:30 GMT

Hi @ArturOA ,

I think you're wrong here. Let's have a look at below screenshot. I'm able to add permission to ADF managed identity to Serveless Warehouse. You can also create group and put service principal/managed identity inside this group and give permission to entire group.

Re: Attaching to Serverless from Azure Data Factory via Service Principal

JakubSkibicki — Wed, 06 Nov 2024 10:51:21 GMT

@ArturOA Have you synced this Managed Identity of ADF as SPN to Databricks?

Re: Attaching to Serverless from Azure Data Factory via Service Principal

h_h_ak — Thu, 07 Nov 2024 22:01:53 GMT

Does the service principal has access and permission for the notebook?