topic Spark streaming is not able to assume role in Data Engineering https://community.databricks.com/t5/data-engineering/spark-streaming-is-not-able-to-assume-role/m-p/45641#M27960 <P>Hello,</P><P>I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a&nbsp;<SPAN>403 error.&nbsp;&nbsp;</SPAN>The code is provided below:</P><DIV><DIV><DIV><PRE><SPAN>spark</SPAN>.readStream<BR /> .format(<SPAN>"s3-sqs"</SPAN>)<BR /> .option(<SPAN>"fileFormat"</SPAN>, <SPAN>"json"</SPAN>)<BR /> .option(<SPAN>"roleArn"</SPAN>, roleArn)<BR /> .option(<SPAN>"compression"</SPAN>, <SPAN>"gzip"</SPAN>)<BR /> .option(<SPAN>"queueUrl"</SPAN>, sqsUrl)<BR /> .option(<SPAN>"badRecordsPath"</SPAN>, badRecordsPath)<BR /> .option(<SPAN>"maxFilesPerTrigger"</SPAN>, <SPAN>2500</SPAN>)<BR /> .schema(schema)<BR /> .load</PRE></DIV></DIV><DIV>&nbsp;</DIV><DIV><SPAN>However, the code below is working perfectly fine with the auto-loader</SPAN></DIV><DIV><DIV><PRE><SPAN>spark</SPAN>.readStream<BR /> .format(<SPAN>"cloudFiles"</SPAN>)<BR /> .option(<SPAN>"cloudFiles.roleArn"</SPAN>, roleArn)<BR /> .option(<SPAN>"cloudFiles.format"</SPAN>, <SPAN>"json"</SPAN>)<BR /> .option(<SPAN>"cloudFiles.maxFileAge"</SPAN>, <SPAN>"14 days"</SPAN>)<BR /> .option(<SPAN>"cloudFiles.queueUrl"</SPAN>, sqsUrl)<BR /> .option(<SPAN>"cloudFiles.maxFilesPerTrigger"</SPAN>, <SPAN>2500</SPAN>)<BR /> .option(<SPAN>"badRecordsPath"</SPAN>, badRecordsPath)<BR /> .schema(schema)<BR /> .load</PRE><P>Can you please support me here?</P></DIV></DIV></DIV> Fri, 22 Sep 2023 09:55:25 GMT Vaibhav1000 2023-09-22T09:55:25Z Spark streaming is not able to assume role https://community.databricks.com/t5/data-engineering/spark-streaming-is-not-able-to-assume-role/m-p/45641#M27960 <P>Hello,</P><P>I am trying to assume an IAM role in spark streaming with "s3-sqs" format. It is giving a&nbsp;<SPAN>403 error.&nbsp;&nbsp;</SPAN>The code is provided below:</P><DIV><DIV><DIV><PRE><SPAN>spark</SPAN>.readStream<BR /> .format(<SPAN>"s3-sqs"</SPAN>)<BR /> .option(<SPAN>"fileFormat"</SPAN>, <SPAN>"json"</SPAN>)<BR /> .option(<SPAN>"roleArn"</SPAN>, roleArn)<BR /> .option(<SPAN>"compression"</SPAN>, <SPAN>"gzip"</SPAN>)<BR /> .option(<SPAN>"queueUrl"</SPAN>, sqsUrl)<BR /> .option(<SPAN>"badRecordsPath"</SPAN>, badRecordsPath)<BR /> .option(<SPAN>"maxFilesPerTrigger"</SPAN>, <SPAN>2500</SPAN>)<BR /> .schema(schema)<BR /> .load</PRE></DIV></DIV><DIV>&nbsp;</DIV><DIV><SPAN>However, the code below is working perfectly fine with the auto-loader</SPAN></DIV><DIV><DIV><PRE><SPAN>spark</SPAN>.readStream<BR /> .format(<SPAN>"cloudFiles"</SPAN>)<BR /> .option(<SPAN>"cloudFiles.roleArn"</SPAN>, roleArn)<BR /> .option(<SPAN>"cloudFiles.format"</SPAN>, <SPAN>"json"</SPAN>)<BR /> .option(<SPAN>"cloudFiles.maxFileAge"</SPAN>, <SPAN>"14 days"</SPAN>)<BR /> .option(<SPAN>"cloudFiles.queueUrl"</SPAN>, sqsUrl)<BR /> .option(<SPAN>"cloudFiles.maxFilesPerTrigger"</SPAN>, <SPAN>2500</SPAN>)<BR /> .option(<SPAN>"badRecordsPath"</SPAN>, badRecordsPath)<BR /> .schema(schema)<BR /> .load</PRE><P>Can you please support me here?</P></DIV></DIV></DIV> Fri, 22 Sep 2023 09:55:25 GMT https://community.databricks.com/t5/data-engineering/spark-streaming-is-not-able-to-assume-role/m-p/45641#M27960 Vaibhav1000 2023-09-22T09:55:25Z