cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

sunkam
by New Contributor
  • 3287 Views
  • 4 replies
  • 0 kudos

Unable to read from azure blob using SAS token\\

I have tried many times all the answers from the internet and stackover flowI have already created the config section before this steps, it passed but this below step is not executing.

image
  • 3287 Views
  • 4 replies
  • 0 kudos
Latest Reply
aockenden
New Contributor III
  • 0 kudos

We were getting this problem when using directory-scoped SAS tokens. While I know there are a number of potential issues that can cause this problem, one potential explanation is that it turns out there is an undocumented spark setting needed on the ...

  • 0 kudos
3 More Replies
vivek_sinha
by Contributor
  • 7737 Views
  • 3 replies
  • 4 kudos

Resolved! Getting Authentication Error while accessing Azure Blob table (wasb) URL using PySpark

I am trying to access the Azure Blob table using Pyspark but getting an Authentication Error. Here I am passing SAS token (HTTP and HTTPS enabled) but it's working only with WASBS (HTTPS) URL, not with WASB (HTTP) URL.Even I tried with Account key as...

  • 7737 Views
  • 3 replies
  • 4 kudos
Latest Reply
vivek_sinha
Contributor
  • 4 kudos

Hi @Arvind Ravish​  The issue got fixed after passing HTTP and HTTPS enabled token to spark executors.Thanks again for your help

  • 4 kudos
2 More Replies
halfwind22
by New Contributor III
  • 10173 Views
  • 9 replies
  • 10 kudos

Resolved! Unable to write csv files to Azure BLOB using pandas to_csv ()

I am using a Py function to read some data from a GET endpoint and write them as a CSV file to a Azure BLOB location.My GET endpoint takes 2 query parameters,param1 and param2. So initially, I have a dataframe paramDf that has two columns param1 and ...

  • 10173 Views
  • 9 replies
  • 10 kudos
Latest Reply
halfwind22
New Contributor III
  • 10 kudos

@Hubert Dudek​ I cant issue a spark command to executor node, throws up an error ,because foreach distributes the processing.

  • 10 kudos
8 More Replies
Labels