Is there a way to get some diagnostic information from the underlying libraries? Maybe an environment variable I can pass or set on the cluster level, that would show up in the Spark driver or worker logs?

I think that stack trace is from the old Azure storage library (com.microsoft.azure.storage). I'd like to know what endpoint it's calling, and where it's getting a token from (and what kind).

The external location has an associated storage credential with a managed system identity. How does the token from that MSI get provided to the classic compute cluster? Typically the Azure storage SDK would connect to the IMDS endpoint and get an access token based on the virtual machine's managed system identity. But here the MSI is tied to the storage credential object, not the VM. That doesn't seem to be a problem for serverless compute, so maybe there's something in the control plane that is getting a token and passing it through to the cluster.