Hi There,

I have been trying to create an external table on Azure Databricks with below statement.

df.write.partitionBy("year", "month", "day").format('org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat').option("path",sourcepath).mode("overwrite").saveAsTable('test.table_test')

sourcepath above is the path on ADLS storage account.

the code works and the table is created pointing to the adlas storage account container, but I get following exception

analysisexception: org.apache.hadoop.hive.ql.metadata.hiveexception: metaexception(message:got exception: shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.keyproviderexception failure to initialize configuration

I also tried setting up configs at cluster level as suggested on

https://community.databricks.com/s/question/0D58Y00009EV8pCSAT/external-table-issue-format-in-databr...

I configured the cluster level configs as below

fs.azure.account.oauth2.client.endpoint.<STORAGE ACCOUNT>.dfs.core.windows.net https://login.microsoftonline.com/<key>/oauth2/token

fs.azure.account.auth.type.<STORAGE ACCOUNT>.dfs.core.windows.net OAuth

fs.azure.account.oauth2.client.id.<STORAGE ACCOUNT>.dfs.core.windows.net ClientId

fs.azure.account.oauth.provider.type.<STORAGE ACCOUNT>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider

fs.azure.account.oauth2.client.secret.<STORAGE ACCOUNT>.dfs.core.windows.net {{secrets/scope/Secret_Key_Name}}

But it still does not work and throws the same exception.

Can anyone who may have encountered and resolved this issue earlier guide me on if I am missing any configs above and what can be the reason I am getting above excpetion although the data is getting written to adls and a table is getting created pointing to the same data.

Thanks