Spark streaming is not able to assume role

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hello,

I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a 403 error. The code is provided below:

spark.readStream
  .format("s3-sqs")
  .option("fileFormat", "json")
  .option("roleArn", roleArn)
  .option("compression", "gzip")
  .option("queueUrl", sqsUrl)
  .option("badRecordsPath", badRecordsPath)
  .option("maxFilesPerTrigger", 2500)
  .schema(schema)
  .load

However, the code below is working perfectly fine with the auto-loader

spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.roleArn", roleArn)
  .option("cloudFiles.format", "json")
  .option("cloudFiles.maxFileAge", "14 days")
  .option("cloudFiles.queueUrl", sqsUrl)
  .option("cloudFiles.maxFilesPerTrigger", 2500)
  .option("badRecordsPath", badRecordsPath)
  .schema(schema)
  .load