cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 
Help
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 

Failed to Access Azure Storage Container from Databricks in ADF Pipeline

p_a_l
Visitor

3 hours ago

 

Hi Databricks community,

I'm new to Databricks and I'm trying to connect Databricks to Azure MySQL using Azure Data Factory.

When I run the pipeline, I get the following error:

 
ErrorCode=AzureDatabricksCommandError, Hit an error when running the command in Azure Databricks. Error details: SparkConnectGrpcException: (shaded.databricks.org.apache.hadoop.fs.azure.AzureException) shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container adfstagedcommandtempdata in account stdeltapipelinesnowflake.blob.core.windows.net not found, and we can't create it using anonymous credentials, and no credentials found for them in the configuration. JVM stacktrace: shaded.databricks.org.apache.hadoop.fs.azure.AzureException Caused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container adfstagedcommandtempdata in account stdeltapipelinesnowflake.blob.core.windows.net not found, and we can't create it using anonymous credentials, and no credentials found for them in the configuration. Caused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container adfstagedcommandtempdata in account stdeltapipelinesnowflake.blob.core.windows.net not found, and we can't create it using anonymous credentials, and no credentials found for them in the configuration.

I'm using stdeltapipelinesnowflake as my Azure Storage Account.

After following the suggestion from the Databricks Assistant and setting the Spark config for the storage account, I now get this:

 

 
[CONFIG_NOT_AVAILABLE] Configuration spark.hadoop.fs.azure.account.key.stdeltapipelinesnowflake.dfs.core.windows.net is not available. SQLSTATE: 42K0I

Has anyone encountered this before and can help me out?

Thanks!

0 Kudos
1 REPLY 1

Coffee77
Contributor III

2 hours ago

Could you provide more details about high level flow or design you are planning? Not understanding completely but you can take a look here how to connect ADF with Databricks via PATs -> https://www.sqlshack.com/integrating-azure-databricks-with-azure-data-factory/ 

 


Lifelong Learner Cloud & Data Solution Architect | https://www.youtube.com/@CafeConData
0 Kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now
Announcements