- 2613 Views
- 3 replies
- 0 kudos
Resolved! AmazonS3 with Autoloader consume "too many" requests or maybe not!
After successfully loading 3 small files (2 KB each) in from AWS S3 using Auto Loader for learning purposes, I got, few hours later, a "AWS Free tier limit alert", although I haven't used the AWS account for a while.Does this streaming service on Da...
- 2613 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi, ​​Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. Auto Loader can load data files from AWS S3 (s3://), Azure Data Lake Storage Gen2 (ADLS Gen2, abfss://), Google Cloud Storage (GCS, gs://), Azur...
- 0 kudos
- 2460 Views
- 2 replies
- 2 kudos
Resolved! create a dataframe with all the responses from the api requests within foreachPartition
I am trying to execute an api call to get an object(json) from amazon s3 and I am using foreachPartition to execute multiple calls in paralleldf.rdd.foreachPartition(partition => { //Initialize list buffer var buffer_accounts1 = new ListBuffer[St...
- 2460 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Sandesh Puligundla​ ,Thank you for sharing the solution. We will mark it as "best" response so, in the future is another user has the same question, they will be able to find the solution right away.
- 2 kudos