I wanted to provide an update on this issue and see if the added information can help resolve the problem. So, after doing some research, it turns out that the Unity Catalog cluster is unable to reach the Azure Instance Metadata Service (IMDS) possibly due to some network restrictions for unity catalog enabled clusters. This prevents the cluster from getting an access token that is needed to use Managed Identity. I used curl to hit the IMDS endpoint and it errors out in case of Unity Catalog.

%sh

curl -H Metadata:true http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://databa...

I have raised this with our Databricks administrators but in order to help them out, does anyone know where these networking settings might be for Unity Catalog where this IP/port can be whitelisted?

Thanks,

Rushi