Hi,

actually we were able to fix the issue, but I completely forgot about my post.
The problem was, that some Databricks URLs were not accessible from the cluster due to firewall restrictions.
We had to make sure that all URLs from this list for the region our cluster is in was reachable: IP addresses and domains for Azure Databricks services and assets - Azure Databricks | Microsoft Lea...

This fixed the issue and the cluster now runs normally.