topic Re: ADF - Fails to run job in workspace in another subscription in Get Started Discussions

ADF - Fails to run job in workspace in another subscription

IvanK — Thu, 27 Jun 2024 09:26:46 GMT

We are trying to run a Databricks job from ADF, but we keep getting the following error:

Operation on target Run dbt job failed: {"error_code":"PERMISSION_DENIED","message":"User <adf-mi-prod-id> does not have Manage Run or Owner or Admin permissions on job 123"}.

The managed identity that should run the job, adf-mi-prod, has permission "Manage" Run on job 123 (I have also tried giving it permission "Is Owner").

When the job is triggered directly in Databricks, it runs normally, but when ADF is trying to trigger it, it fails.

Does anyone know why triggering it through ADF is failing?

Extra info

We are using ADF for orchestration.

ADF instance lies in our prod subscription and we have Databricks workspaces in both test and prod subscription.

We have the same job in both workspaces. ADF succeeds to run the job in prod workspace, but fails to run it in test workspace.

The ADF MI has the same permissions on the clusters and SQL warehouses in both workspaces.

"Run as" on the job is set to adf-mi-prod for both jobs.

Permission "Can manage" are set for adf-mi-prod on both jobs.

Re: ADF - Fails to run job in workspace in another subscription

IvanK — Fri, 28 Jun 2024 12:31:35 GMT

Hello @Retired_mod ,

Sure, this is what I can retrieve from ADF:
Operation on target Run dbt job failed: {"error_code":"PERMISSION_DENIED","message":"User <adf-mi-prod-id> does not have Manage Run or Owner or Admin permissions on job 123"}

<adf-mi-prod-id> is the Client ID of the MI.

I can not see any job runs of job 123 in Databricks, whenever the job is triggered by ADF and I get the error message above.

If I trigger the job manually in Databricks, the job run succeeds (and I see a job run in the UI)

Re: ADF - Fails to run job in workspace in another subscription

riquedan — Mon, 01 Jul 2024 07:14:57 GMT

I am seeing the error message above whenever job 123 is triggered by ADF, but I can't find any instances of this job running in Databricks.

Re: ADF - Fails to run job in workspace in another subscription

IvanK — Mon, 01 Jul 2024 10:09:59 GMT

Hello @Retired_mod

Thank you for the suggestions! However, they did not work.

The odd thing is that when I run the job manually in Databricks, it works. It is only when I try to run the job through ADF that it fails.

Any other idea as to why this is happening and how to solve this?

PS to clarify the setup:

We have 2 Databricks workspaces in 2 different Azure subscriptions: test and prod.

ADF lies in prod subscription and uses a managed identity. The managed identity has been added to both workspaces, and have identical permissions/grants/priviliges in both workspaces.

Here is a comparison matrix of when the job is run manually or through ADF in both workspaces:

	Test workspace	Prod workspace
Run job through ADF	FAILS	WORKS
Run job Manually	WORKS	WORKS