cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Leo_138525
by New Contributor II
  • 1921 Views
  • 4 replies
  • 1 kudos

Resolved! RDD not picking up spark configuration for azure storage account access

I want to open some CSV files as an RDD, do some processing and then load it as a DataFrame. Since the files are stored in an Azure blob storage account I need to configure the access accordingly, which for some reason does not work when using an RDD...

  • 1921 Views
  • 4 replies
  • 1 kudos
Latest Reply
Leo_138525
New Contributor II
  • 1 kudos

I decided to load the files into a DataFrame with a single column and then do the processing before splitting it into separate columns and this works just fine.@Hyper Guy​ thanks for the link, I didn't try that but it seems like it would resolve the ...

  • 1 kudos
3 More Replies
KVNARK
by Honored Contributor II
  • 494 Views
  • 1 replies
  • 5 kudos

accessing secret from spark cluster.

passing spark configuration to access blob, adls from data factory while creating job clusterit's working fine, but when in the property we are accessing secret it's not workingspark.hadoop.fs.azure.account.auth.type.{{secrets/scope/key}}.dfs.core.wi...

  • 494 Views
  • 1 replies
  • 5 kudos
Latest Reply
sher
Valued Contributor II
  • 5 kudos

check here : https://docs.databricks.com/security/secrets/secrets.html

  • 5 kudos
Labels