Hello,
I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a 403 error. The code is provided below:
spark.readStream
.format("s3-sqs")
.option("fileFormat", "json")
.option("roleArn", roleArn)
.option("compression", "gzip")
.option("queueUrl", sqsUrl)
.option("badRecordsPath", badRecordsPath)
.option("maxFilesPerTrigger", 2500)
.schema(schema)
.load
However, the code below is working perfectly fine with the auto-loader
spark.readStream
.format("cloudFiles")
.option("cloudFiles.roleArn", roleArn)
.option("cloudFiles.format", "json")
.option("cloudFiles.maxFileAge", "14 days")
.option("cloudFiles.queueUrl", sqsUrl)
.option("cloudFiles.maxFilesPerTrigger", 2500)
.option("badRecordsPath", badRecordsPath)
.schema(schema)
.load
Can you please support me here?