I get the following error when getting a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databrciks.
Failure to initialize configuration for storage account AAAAA.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key
Databricks Workspace:
- Unity catalog is enabled.
- There is a catalog named "Exploration"
- Service principal has "ALL PRIVILEGES" grant on this catalog
Azure storage account: ADLS Gen2
Service principal has Read/Write/Execute access to the root folder. Also, it has the same access to the sub-folder "test".
What should I do to be able to get a list of files?
Notes:
- I am able to create sub-folders in the Exploration storage location.
- Also, I can upload files into the storage location.
- I can get the list of files in the "raw" container using dbutils:
Configurations to access the Azure data lake storage are done by another company. I think they have used "External Locations" in Databricks.