This appears to be a long-standing issue with Azure and Databricks. I found this stack overflow post from a few months back discussing the same issue with no outcome. ManagedIdentityCredential in Shared Compute - Databricks - Stack Overflow. I suspect it is either a glitch or a security override since a shared cluster can be used by multiple users. Hopefully someone from Databricks end can look into this issue further.