I had the same problem, but was connecting via RDD API (I was using my own java adaptor to azure blobs). The https://docs.databricks.com/spark/latest/data-sources/azure/azure-storage.html page states at the bottom that you should create your cluster...
