Connecting to ADLS azure storage and reading csv file from adls

Anwitha — Sat, 22 Jun 2024 05:28:36 GMT

Hi Friends ,

How are you all? I am Anwitha. I am a newbie in azure data bricks community. I am stuck with this error while practising with azure storage account. Could anyone throw a light on this error and provide a solution?

I tried connecting to order.csv file in retail container in azurestorage from azure databricks workspace , but i got this error,

Here is the python code i used ,

dbutils.fs.ls("abfss://retail@careeritazuredestorage.dfs.core.Windows.net")

ExecutionError: An error occurred while calling o411.ls. : Failure to initialize configuration for storage account careeritazuredestorage.dfs.core.Windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:52) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:682) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:2077) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:269) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:235) at com.databricks.common.filesystem.LokiABFS.initialize(LokiABFS.scala:36) at com.databricks.common.filesystem.LokiFileSystem$.$anonfun$getLokiFS$1(LokiFileSystem.scala:149) at com.databricks.common.filesystem.FileSystemCache.getOrCompute(FileSystemCache.scala:46) at com.databricks.common.filesystem.LokiFileSystem$.getLokiFS(LokiFileSystem.scala:146) at com.databricks.common.filesystem.LokiFileSystem.initialize(LokiFileSystem.scala:211) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at com.databricks.unity.SAM.createDelegate(SAM.scala:215) at com.databricks.unity.SAM.createDelegate$(SAM.scala:208) at com.databricks.unity.ClusterDefaultSAM$.createDelegate(SAM.scala:250) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.createDelegate(CredentialScopeFileSystem.scala:85) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.$anonfun$setDelegates$1(CredentialScopeFileSystem.scala:137) at com.databricks.sql.acl.fs.Lazy.apply(DelegatingFileSystem.scala:310) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.getReadDelegate(CredentialScopeFileSystem.scala:98) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.listStatus(CredentialScopeFileSystem.scala:232) at com.databricks.backend.daemon.dbutils.FSUtils.lsWithLimit(DBUtilsCore.scala:294) at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$lsImpl$4(DBUtilsCore.scala:263) at com.databricks.backend.daemon.dbutils.FSUtils.withFsSafetyCheck(DBUtilsCore.scala:150) at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$lsImpl$3(DBUtilsCore.scala:263) at com.databricks.backend.daemon.dbutils.FSUtils.checkPermission(DBUtilsCore.scala:145) at com.databricks.backend.daemon.dbutils.FSUtils.lsImpl(DBUtilsCore.scala:262) at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$ls$1(DBUtilsCore.scala:235) at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:573) at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:669) at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:687) at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216) at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424) at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418) at com.databricks.backend.daemon.dbutils.FSUtils.withAttributionContext(DBUtilsCore.scala:72) at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472) at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455) at com.databricks.backend.daemon.dbutils.FSUtils.withAttributionTags(DBUtilsCore.scala:72) at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:664) at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582) at com.databricks.backend.daemon.dbutils.FSUtils.recordOperationWithResultTags(DBUtilsCore.scala:72) at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:573) at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:542) at com.databricks.backend.daemon.dbutils.FSUtils.recordOperation(DBUtilsCore.scala:72) at com.databricks.backend.daemon.dbutils.FSUtils.recordDbutilsFsOp(DBUtilsCore.scala:136) at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala:235) at sun.reflect.GeneratedMethodAccessor420.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at py4j.Gateway.invoke(Gateway.java:306) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750) Caused by: Invalid configuration value detected for fs.azure.account.key at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.diagnostics.ConfigurationBasicValidator.validate(ConfigurationBasicValidator.java:49) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.diagnostics.Base64StringConfigurationBasicValidator.validate(Base64StringConfigurationBasicValidator.java:40) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.validateStorageAccountKey(SimpleKeyProvider.java:71) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:49) ... 58 more

Re: Connecting to ADLS azure storage and reading csv file from adls

Anwitha — Sat, 22 Jun 2024 05:32:20 GMT

Friends , I ran this code for storage account before this code and it ran without any error.

storage_account_name = 'careeritazuredestorage'

storage_account_access_key = 'snQQGC+EfEs2vrl4XuRhkoA7VJ0j9AokReIEJG/KaiB5sSVTedD9M61BvSCo5jd0VaBSifdgGcKw+AStJgWKXg=='

Re: Connecting to ADLS azure storage and reading csv file from adls

jacovangelder — Sat, 22 Jun 2024 09:17:29 GMT

Ermm, you might want to edit your post and remove that storage account access key!
In combination with the storage account name (that is also in your post), everyone will be able to access it if it isn't behind a firewall!

Btw, you might want to study storage mounts and how they work. Right now you are passing environment variables (hardcoded too). Not the best way. https://docs.databricks.com/en/dbfs/mounts.html

topic Re: Connecting to ADLS azure storage and reading csv file from adls in Data Engineering

Connecting to ADLS azure storage and reading csv file from adls

Re: Connecting to ADLS azure storage and reading csv file from adls

Re: Connecting to ADLS azure storage and reading csv file from adls