User: anonymous is not authorized to perform: sqs:receivemessage on resource

rt-slowth — Thu, 11 Jan 2024 06:15:03 GMT

from pyspark.sql import functions as F from pyspark.sql import types as T from pyspark.sql import DataFrame, Column from pyspark.sql.types import Row import dlt S3_PATH = 's3://datalake-lab/xxxx/' S3_SCHEMA = 's3://datalake-lab/xxxx/schemas/' @dlt.table def test_raw(): raw_df = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "parquet") .option("cloudFiles.schemaLocation", f"{S3_SCHEMA}") .option("cloudFiles.useNotifications", "true") .option("cloudFiles.region", "ap-northeast-1") .option("cloudFile.roleArn", "<Instance Profile ARN>") .option( "cloudFiles.queueUrl", "https://sqs.<region>.amazonaws.com/<account-id>/databricks-auto-ingest-demo", ) .load(f"{S3_PATH}") # <-- Specify the path to the gzip files here # .selectExpr("*", '_metadata.file_name as input_file_name', '_metadata.file_path as input_file_path') ) return raw_df

com.amazonaws.services.sqs.model.AmazonSQSException: User: anonymous is not authorized to perform: sqs:receivemessage on resource: arn:aws:sqs:<region>:<account-id>:databricks-auto-ingest-demo because no resource-based policy allows the sqs:receivemessage action (Service: AmazonSQS; Status Code: 403;

What settings in AWS should I check?

Re: User: anonymous is not authorized to perform: sqs:receivemessage on resource

rt-slowth — Fri, 12 Jan 2024 08:38:25 GMT

Hi, @Retired_mod

This time I ran into a different error. The SQS service is not found.
Here's what I'm wondering

1. I set the IAM policy in the file notification mode guide in autoloader to the S3 instance profile, how should I write the policy for SQS?

topic Re: User: anonymous is not authorized to perform: sqs:receivemessage on resource in Data Engineering

User: anonymous is not authorized to perform: sqs:receivemessage on resource

Re: User: anonymous is not authorized to perform: sqs:receivemessage on resource