Spark streaming is not able to assume role

[Vaibhav1000]

Hello,

I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a 403 error.  The code is provided below:

spark.readStream
  .format("s3-sqs")
  .option("fileFormat", "json")
  .option("roleArn", roleArn)
  .option("compression", "gzip")
  .option("queueUrl", sqsUrl)
  .option("badRecordsPath", badRecordsPath)
  .option("maxFilesPerTrigger", 2500)
  .schema(schema)
  .load

However, the code below is working perfectly fine with the auto-loader

spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.roleArn", roleArn)
  .option("cloudFiles.format", "json")
  .option("cloudFiles.maxFileAge", "14 days")
  .option("cloudFiles.queueUrl", sqsUrl)
  .option("cloudFiles.maxFilesPerTrigger", 2500)
  .option("badRecordsPath", badRecordsPath)
  .schema(schema)
  .load

Can you please support me here?

[Kaniz_Fatma]

Hi @Vaibhav1000, The 403 error you're encountering with the "s3-sqs" format could be due to incorrect configuration or insufficient permissions.

You can try the following steps to resolve this issue:

1. Check your IAM role permissions: Ensure that the IAM role you're trying to assume has the necessary permissions to access the required resources.

2. Configure the IAM role correctly: You may need to set the IAM role for the "s3-sqs" format using the Hadoop configuration, similar to the method mentioned in the provided information.