We are trying to run a Databricks job from ADF, but we keep getting the following error:
Operation on target Run dbt job failed: {"error_code":"PERMISSION_DENIED","message":"User <adf-mi-prod-id> does not have Manage Run or Owner or Admin permissions on job 123"}.
The managed identity that should run the job, adf-mi-prod, has permission "Manage" Run on job 123 (I have also tried giving it permission "Is Owner").
When the job is triggered directly in Databricks, it runs normally, but when ADF is trying to trigger it, it fails.
Does anyone know why triggering it through ADF is failing?
Extra info
We are using ADF for orchestration.
ADF instance lies in our prod subscription and we have Databricks workspaces in both test and prod subscription.
We have the same job in both workspaces. ADF succeeds to run the job in prod workspace, but fails to run it in test workspace.
The ADF MI has the same permissions on the clusters and SQL warehouses in both workspaces.
"Run as" on the job is set to adf-mi-prod for both jobs.
Permission "Can manage" are set for adf-mi-prod on both jobs.