@Wojciech_BUK  thanks a lot for all your advice! I solve the issue after all, and the solution was ridiculously easy as well as ridiculously not obvious (to me at least).

So the thing was that I was adding my Data Factory managed identity to the External Location permissions using it's display name. And all I had to do instead is I had to add it using it's "Managed Identity Application ID".. 

I am not sure if that's default behavior or if I configured something wrong somewhere else to come to this. But anyway now it works!