cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
sunkam
by • New Contributor
  • 3823 Views
  • 4 replies
  • 0 kudos

Unable to read from azure blob using SAS token\\

I have tried many times all the answers from the internet and stackover flowI have already created the config section before this steps, it passed but this below step is not executing.

image
Data Engineering
Reply
  • 3823 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
aockenden
New Contributor III
  • 0 kudos

We were getting this problem when using directory-scoped SAS tokens. While I know there are a number of potential issues that can cause this problem, one potential explanation is that it turns out there is an undocumented spark setting needed on the ...

  • 0 kudos
3 More Replies
vivek_sinha
by • Contributor
  • 8723 Views
  • 3 replies
  • 4 kudos

Resolved! Getting Authentication Error while accessing Azure Blob table (wasb) URL using PySpark

I am trying to access the Azure Blob table using Pyspark but getting an Authentication Error. Here I am passing SAS token (HTTP and HTTPS enabled) but it's working only with WASBS (HTTPS) URL, not with WASB (HTTP) URL.Even I tried with Account key as...

Data Engineering
Reply
  • 8723 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
vivek_sinha
Contributor
  • 4 kudos

Hi @Arvind Ravish​  The issue got fixed after passing HTTP and HTTPS enabled token to spark executors.Thanks again for your help

  • 4 kudos
2 More Replies
halfwind22
by • New Contributor III
  • 11321 Views
  • 9 replies
  • 10 kudos

Resolved! Unable to write csv files to Azure BLOB using pandas to_csv ()

I am using a Py function to read some data from a GET endpoint and write them as a CSV file to a Azure BLOB location.My GET endpoint takes 2 query parameters,param1 and param2. So initially, I have a dataframe paramDf that has two columns param1 and ...

Data Engineering
Reply
  • 11321 Views
  • 9 replies
  • 10 kudos
View Replies
Latest Reply
halfwind22
New Contributor III
  • 10 kudos

@Hubert Dudek​ I cant issue a spark command to executor node, throws up an error ,because foreach distributes the processing.

  • 10 kudos
8 More Replies
Labels