cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

ADF - Fails to run job in workspace in another subscription

IvanK
New Contributor III

 We are trying to run a Databricks job from ADF, but we keep getting the following error:

Operation on target Run dbt job failed: {"error_code":"PERMISSION_DENIED","message":"User <adf-mi-prod-id> does not have Manage Run or Owner or Admin permissions on job 123"}.

The managed identity that should run the job, adf-mi-prod, has permission "Manage" Run on job 123 (I have also tried giving it permission "Is Owner").

When the job is triggered directly in Databricks, it runs normally, but when ADF is trying to trigger it, it fails.

Does anyone know why triggering it through ADF is failing?

Extra info

We are using ADF for orchestration.

ADF instance lies in our prod subscription and we have Databricks workspaces in both test and prod subscription.

We have the same job in both workspaces. ADF succeeds to run the job in prod workspace, but fails to run it in test workspace.

The ADF MI has the same permissions on the clusters and SQL warehouses in both workspaces.

"Run as" on the job is set to adf-mi-prod for both jobs.

Permission "Can manage" are set for adf-mi-prod on both jobs.

3 REPLIES 3

IvanK
New Contributor III

Hello @Retired_mod ,

Sure, this is what I can retrieve from ADF:
Operation on target Run dbt job failed: {"error_code":"PERMISSION_DENIED","message":"User <adf-mi-prod-id> does not have Manage Run or Owner or Admin permissions on job 123"}

<adf-mi-prod-id> is the Client ID of the MI.

I can not see any job runs of job 123 in Databricks, whenever the job is triggered by ADF and I get the error message above.

If I trigger the job manually in Databricks, the job run succeeds (and I see a job run in the UI)

IvanK
New Contributor III

Hello @Retired_mod 

Thank you for the suggestions! However, they did not work.

The odd thing is that when I run the job manually in Databricks, it works. It is only when I try to run the job through ADF that it fails.

Any other idea as to why this is happening and how to solve this?

PS to clarify the setup:

We have 2 Databricks workspaces in 2 different Azure subscriptions: test and prod.

ADF lies in prod subscription and uses a managed identity. The managed identity has been added to both workspaces, and have identical permissions/grants/priviliges in both workspaces.

Here is a comparison matrix of when the job is run manually or through ADF in both workspaces:

 Test workspaceProd workspace
Run job through ADFFAILSWORKS
Run job ManuallyWORKSWORKS

riquedan
New Contributor II

I am seeing the error message above whenever job 123 is triggered by ADF, but I can't find any instances of this job running in Databricks.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group