cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

#HelpPost for Azure Blob to Databricks connection.

menonshiji
New Contributor

Hi,

There is a set of .csv/.txt files in the storage container ie Azure Blob Storage/ Azure Storage Gen 2. I would like to ingest the files to Databricks. Dataset,LinkedServices was created on both ends. Also an all purpose cluster was created in Bricks. Can we ingest the data using ETL Tool ADF from Blob to Databricks?

I tried creating and debugging the pipeline, but failed with the following message in the pipeline - "ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container csv-file-storage in account storageaccountfordev.blob.core.windows.net using anonymous credentials, and no credentials found for them in the configuration.
Caused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container csv-file-storage in account storageaccountfordev.blob.core.windows.net using anonymous credentials, and no credentials found for them in the configuration.
Caused by: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Public access is not permitted on this storage account.. "

I have a limited experience with Databricks. Can anyone give me some insights about this? Basically i wanted to know how the ADF works. 

Thanks,

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @menonshiji, The error in the Databricks Pipeline indicates that the Azure Blob Storage container is inaccessible due to the lack of proper credentials. To resolve this issue, make sure to include the necessary credentials in the Spark configuration of your Databricks cluster. 

 

 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!