- 2649 Views
- 4 replies
- 1 kudos
Resolved! RDD not picking up spark configuration for azure storage account access
I want to open some CSV files as an RDD, do some processing and then load it as a DataFrame. Since the files are stored in an Azure blob storage account I need to configure the access accordingly, which for some reason does not work when using an RDD...
- 2649 Views
- 4 replies
- 1 kudos
- 1 kudos
I decided to load the files into a DataFrame with a single column and then do the processing before splitting it into separate columns and this works just fine.@Hyper Guy​ thanks for the link, I didn't try that but it seems like it would resolve the ...
- 1 kudos
- 698 Views
- 1 replies
- 5 kudos
accessing secret from spark cluster.
passing spark configuration to access blob, adls from data factory while creating job clusterit's working fine, but when in the property we are accessing secret it's not workingspark.hadoop.fs.azure.account.auth.type.{{secrets/scope/key}}.dfs.core.wi...
- 698 Views
- 1 replies
- 5 kudos
- 5 kudos
check here : https://docs.databricks.com/security/secrets/secrets.html
- 5 kudos