cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Connecting to ADLS azure storage and reading csv file from adls

Anwitha
New Contributor II

Hi Friends , 

How are you all? I am Anwitha. I am a newbie in azure data bricks community. I am stuck with this error while practising with azure storage account. Could anyone throw a light on this error and provide a solution?

I tried connecting to order.csv file in retail container in azurestorage from azure databricks workspace , but i got this error,

Here is the python code i used ,

 
dbutils.fs.ls("abfss://retail@careeritazuredestorage.dfs.core.Windows.net")
 
ExecutionError: An error occurred while calling o411.ls. : Failure to initialize configuration for storage account careeritazuredestorage.dfs.core.Windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:52) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:682) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:2077) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:269) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:235) at com.databricks.common.filesystem.LokiABFS.initialize(LokiABFS.scala:36) at com.databricks.common.filesystem.LokiFileSystem$.$anonfun$getLokiFS$1(LokiFileSystem.scala:149) at com.databricks.common.filesystem.FileSystemCache.getOrCompute(FileSystemCache.scala:46) at com.databricks.common.filesystem.LokiFileSystem$.getLokiFS(LokiFileSystem.scala:146) at com.databricks.common.filesystem.LokiFileSystem.initialize(LokiFileSystem.scala:211) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3611) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:554) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at com.databricks.unity.SAM.createDelegate(SAM.scala:215) at com.databricks.unity.SAM.createDelegate$(SAM.scala:208) at com.databricks.unity.ClusterDefaultSAM$.createDelegate(SAM.scala:250) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.createDelegate(CredentialScopeFileSystem.scala:85) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.$anonfun$setDelegates$1(CredentialScopeFileSystem.scala:137) at com.databricks.sql.acl.fs.Lazy.apply(DelegatingFileSystem.scala:310) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.getReadDelegate(CredentialScopeFileSystem.scala:98) at com.databricks.sql.acl.fs.CredentialScopeFileSystem.listStatus(CredentialScopeFileSystem.scala:232) at com.databricks.backend.daemon.dbutils.FSUtils.lsWithLimit(DBUtilsCore.scala:294) at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$lsImpl$4(DBUtilsCore.scala:263) at com.databricks.backend.daemon.dbutils.FSUtils.withFsSafetyCheck(DBUtilsCore.scala:150) at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$lsImpl$3(DBUtilsCore.scala:263) at com.databricks.backend.daemon.dbutils.FSUtils.checkPermission(DBUtilsCore.scala:145) at com.databricks.backend.daemon.dbutils.FSUtils.lsImpl(DBUtilsCore.scala:262) at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$ls$1(DBUtilsCore.scala:235) at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:573) at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:669) at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:687) at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:426) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62) at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:216) at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:424) at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418) at com.databricks.backend.daemon.dbutils.FSUtils.withAttributionContext(DBUtilsCore.scala:72) at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472) at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455) at com.databricks.backend.daemon.dbutils.FSUtils.withAttributionTags(DBUtilsCore.scala:72) at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:664) at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:582) at com.databricks.backend.daemon.dbutils.FSUtils.recordOperationWithResultTags(DBUtilsCore.scala:72) at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:573) at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:542) at com.databricks.backend.daemon.dbutils.FSUtils.recordOperation(DBUtilsCore.scala:72) at com.databricks.backend.daemon.dbutils.FSUtils.recordDbutilsFsOp(DBUtilsCore.scala:136) at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala:235) at sun.reflect.GeneratedMethodAccessor420.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at py4j.Gateway.invoke(Gateway.java:306) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750) Caused by: Invalid configuration value detected for fs.azure.account.key at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.diagnostics.ConfigurationBasicValidator.validate(ConfigurationBasicValidator.java:49) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.diagnostics.Base64StringConfigurationBasicValidator.validate(Base64StringConfigurationBasicValidator.java:40) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.validateStorageAccountKey(SimpleKeyProvider.java:71) at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider.getStorageAccountKey(SimpleKeyProvider.java:49) ... 58 more
2 REPLIES 2

Anwitha
New Contributor II

Friends , I ran this code for storage account before this code and it ran without any error.

 

storage_account_name = 'careeritazuredestorage'
storage_account_access_key = 'snQQGC+EfEs2vrl4XuRhkoA7VJ0j9AokReIEJG/KaiB5sSVTedD9M61BvSCo5jd0VaBSifdgGcKw+AStJgWKXg=='

jacovangelder
Contributor III

Ermm, you might want to edit your post and remove that storage account access key! 
In combination with the storage account name (that is also in your post), everyone will be able to access it if it isn't behind a firewall! 

Btw, you might want to study storage mounts and how they work. Right now you are passing environment variables (hardcoded too). Not the best way. https://docs.databricks.com/en/dbfs/mounts.html

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!