I had the same issue with using dynamic partitioning in ADLS using Databricks Spark Sql.

You need to pass ADLS configs as Spark configs during cluster creation:

dfs.adls.oauth2.client.id ***

dfs.adls.oauth2.refresh.url https://login.microsoftonline.com/**/oauth2/token

dfs.adls.oauth2.credential **

dfs.adls.oauth2.access.token.provider.type ClientCredential

dfs.adls.oauth2.access.token.provider org.apache.hadoop.fs.adls.oauth2.ConfCredentialBasedAccessTokenProvider

You also need to set hadoopConfiguration for RDD related functionality:

spark.sparkContext.hadoopConfiguration.set("dfs.adls.oauth2.access.token.provider.type", spark.conf.get("dfs.adls.oauth2.access.token.provider.type"))

spark.sparkContext.hadoopConfiguration.set("dfs.adls.oauth2.client.id", spark.conf.get("dfs.adls.oauth2.client.id"))

spark.sparkContext.hadoopConfiguration.set("dfs.adls.oauth2.credential", spark.conf.get("dfs.adls.oauth2.credential"))

spark.sparkContext.hadoopConfiguration.set("dfs.adls.oauth2.refresh.url", spark.conf.get("dfs.adls.oauth2.refresh.url"))

Those two measures fixed the issue for me.

/Taras