Hi,
We have issues trying to run Databricks notebooks orchestrated with Azure Data Factory. We have been doing this for a while now without any issues when we use Job Clusters, Existing General Purpose Clusters, or Cluster Pools. We use an Azure Data Factory Managed Service Identity (service principal) that we have integrated into our Databricks workspace.
The problem is when we try to use an existing Serverless SQL Warehouse. We are able to get the ID and all necessary parameters. When we test the connection it is successful. However, we are not able to run the notebook. We get the error:
"Run aborted because the job run-as lacks Attach permissions on the underlying cluster"
As shown below.
However, I am able to run the same notebook successfully when I use my PAT to connect to the Serverles warehouse.
Any idea on how to solve the issue? We really don't want to run our jobs based on personal credentials...
