Best option is to use "External Locations" and "Storage Credentials" under "Unity Catalog". This avoids tons of problems.

If Unity Catalog is not possible, only way to achieve this I was able to verify is:

When you deploy a Databricks workspace, a user-assigned dbManagedIdentity is created in background. Not sure if this only happens in non-VNET injected workspaces or for all setups (pending to check)

Previous dbManagedIdentity is then associated to VMs making up the cluster. So, if you assign proper roles to this managed identity over ADLS, it turns out DefaultAzureCredentials gets this "identity" (checked in access token payload and code on my own) and the code pasted above really works. VMs are using this managed identity to access ADLS 🙂 but (I think) nothing to do with Azure Databricks Connector.

Check here how user-assigned managed identity is assigned to VMs

Kind Regards.