ADF - Fails to run job in workspace in another subscription

[IvanK]

 We are trying to run a Databricks job from ADF, but we keep getting the following error:

Operation on target Run dbt job failed: {"error_code":"PERMISSION_DENIED","message":"User <adf-mi-prod-id> does not have Manage Run or Owner or Admin permissions on job 123"}.

The managed identity that should run the job, adf-mi-prod, has permission "Manage" Run on job 123 (I have also tried giving it permission "Is Owner").

When the job is triggered directly in Databricks, it runs normally, but when ADF is trying to trigger it, it fails.

Does anyone know why triggering it through ADF is failing?

Extra info

We are using ADF for orchestration.

ADF instance lies in our prod subscription and we have Databricks workspaces in both test and prod subscription.

We have the same job in both workspaces. ADF succeeds to run the job in prod workspace, but fails to run it in test workspace.

The ADF MI has the same permissions on the clusters and SQL warehouses in both workspaces.

"Run as" on the job is set to adf-mi-prod for both jobs.

Permission "Can manage" are set for adf-mi-prod on both jobs.

[Kaniz_Fatma]

Hi @IvanK, Could you please share the error stack with us?

[IvanK]

Hello @Kaniz_Fatma ,

Sure, this is what I can retrieve from ADF:
Operation on target Run dbt job failed: {"error_code":"PERMISSION_DENIED","message":"User <adf-mi-prod-id> does not have Manage Run or Owner or Admin permissions on job 123"}

<adf-mi-prod-id> is the Client ID of the MI.

I can not see any job runs of job 123 in Databricks, whenever the job is triggered by ADF and I get the error message above.

If I trigger the job manually in Databricks, the job run succeeds (and I see a job run in the UI)

[Kaniz_Fatma]

Hi @IvanK, The error message you received indicates that the user or service principal running the Databricks job lacks the necessary read permissions on the Azure ADLS storage containers.

Here are some steps to address this issue:

1. Check Permissions on Azure ADLS Storage Containers:

   • Ensure that the user or service principal has been granted the appropriate permissions to access the storage containers. You can do this through the Azure portal or using Azure CLI.
   • Verify that the credentials for accessing the storage containers are correctly configured in the Databricks workspace. Check the Databricks Secret Scopes to ensure the correct secrets are being used and that they haven’t expired.

2. Unity Catalog Considerations:

   • If you recently attached the Unity metastore to your workspace, make sure you’ve created the necessary storage credentials and configured the storage locations.
   • Specifically, grant the “READ FILES” permission over the External Location for the Service Principal executing the read operation from your Storage Account.

3. Job Code and Permissions:

   • Review the job code to ensure that the correct path and permissions are being used when accessing the storage containers. Specify the correct directory or file permissions as needed.

4. Cluster Privileges:

   • Sometimes, job failures can occur due to cluster-related permissions. If the job owner’s privilege to start the cluster is revoked, even an Admin user’s submission can fa...¹. Make sure the job owner has the necessary privileges.

5. Monitoring and Alerts:

   • Consider setting up monitoring and alerts for your Databricks jobs. This way, you’ll receive notifications when issues arise, such as permission errors or job failures.

Remember that troubleshooting permissions can be complex, but these steps should help you identify and resolve the issue. If you’ve tried everything and are still facing problems, feel free to ask for further assistance! 😊

[IvanK]

Hello @Kaniz_Fatma 

Thank you for the suggestions! However, they did not work.

The odd thing is that when I run the job manually in Databricks, it works. It is only when I try to run the job through ADF that it fails.

Any other idea as to why this is happening and how to solve this?

PS to clarify the setup:

We have 2 Databricks workspaces in 2 different Azure subscriptions: test and prod.

ADF lies in prod subscription and uses a managed identity. The managed identity has been added to both workspaces, and have identical permissions/grants/priviliges in both workspaces.

Here is a comparison matrix of when the job is run manually or through ADF in both workspaces:

	Test workspace	Prod workspace
Run job through ADF	FAILS	WORKS
Run job Manually	WORKS	WORKS

[riquedan]

I am seeing the error message above whenever job 123 is triggered by ADF, but I can't find any instances of this job running in Databricks.