Hello @Retired_mod 

Thank you for the suggestions! However, they did not work.

The odd thing is that when I run the job manually in Databricks, it works. It is only when I try to run the job through ADF that it fails.

Any other idea as to why this is happening and how to solve this?

PS to clarify the setup:

We have 2 Databricks workspaces in 2 different Azure subscriptions: test and prod.

ADF lies in prod subscription and uses a managed identity. The managed identity has been added to both workspaces, and have identical permissions/grants/priviliges in both workspaces.

Here is a comparison matrix of when the job is run manually or through ADF in both workspaces: