- 3999 Views
- 3 replies
- 2 kudos
Task not serializable: java.io.NotSerializableException: org.apache.spark.sql.streaming.DataStreamWriter
I have a getS3Object function to get (json) objects located in aws s3 object client_connect extends Serializable { val s3_get_path = "/dbfs/mnt/s3response" def getS3Objects(s3ObjectName: String, s3Client: AmazonS3): String = { val...
- 3999 Views
- 3 replies
- 2 kudos
- 2 kudos
Hey there @Sandesh Puligundla​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear f...
- 2 kudos
- 1749 Views
- 3 replies
- 2 kudos
Resolved! log error to cosmos db
Objective:- Retrieve objects from an S3 bucket using a 'get' api call, write the retrieved object to azure datalake and in case of errors like 404s (object not found) write the error message to cosmos DB"my_dataframe" consists of the a column (s3Obje...
- 1749 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Sandesh Puligundla​ issue is that you are using spark context inside foreachpartition. You can create a dataframe only on the spark driver. Few stack overflow references https://stackoverflow.com/questions/46964250/nullpointerexception-creatin...
- 2 kudos
- 1288 Views
- 0 replies
- 0 kudos
- 1288 Views
- 0 replies
- 0 kudos