User: anonymous is not authorized to perform: sqs:receivemessage on resource

[rt-slowth]

from pyspark.sql import functions as F
from pyspark.sql import types as T
from pyspark.sql import DataFrame, Column
from pyspark.sql.types import Row
import dlt

S3_PATH = 's3://datalake-lab/xxxx/'
S3_SCHEMA = 's3://datalake-lab/xxxx/schemas/'

@dlt.table
def test_raw():
    raw_df = (
        spark.readStream.format("cloudFiles")
        .option("cloudFiles.format", "parquet")
        .option("cloudFiles.schemaLocation", f"{S3_SCHEMA}")
        .option("cloudFiles.useNotifications", "true")
        .option("cloudFiles.region", "ap-northeast-1")
        .option("cloudFile.roleArn", "<Instance Profile ARN>")
        .option(
            "cloudFiles.queueUrl",
            "https://sqs.<region>.amazonaws.com/<account-id>/databricks-auto-ingest-demo",
        )
        .load(f"{S3_PATH}")  # <-- Specify the path to the gzip files here
        # .selectExpr("*", '_metadata.file_name as input_file_name', '_metadata.file_path as input_file_path')
    )

    return raw_df

com.amazonaws.services.sqs.model.AmazonSQSException: User: anonymous is not authorized to perform: sqs:receivemessage on resource: arn:aws:sqs:<region>:<account-id>:databricks-auto-ingest-demo because no resource-based policy allows the sqs:receivemessage action (Service: AmazonSQS; Status Code: 403;

What settings in AWS should I check?

[Kaniz]

Hi @rt-slowth, Check the IAM Role/User: Ensure that the IAM role or user associated with your application has the n....

Update the IAM Policy: If the necessary permissions are not present, you will need to update the IAM policy attached to your role or user. The policy should include the sqs:receivemessage action and the ARN of the SQS queue as the resource.

Check Resource-Based Policies: If a resource-based policy is attached to your SQS queue, ensure that....

Check for Explicit Deny: Ensure that there are no policies that explicitly deny the sqs:receivemessa....

[rt-slowth]

Hi, @Kaniz 

This time I ran into a different error. The SQS service is not found.
Here's what I'm wondering

1. I set the IAM policy in the file notification mode guide in autoloader to the S3 instance profile, how should I write the policy for SQS?

[Kaniz]

Hi @rt-slowth, The IAM policy for SQS should allow the necessary actions on the specific SQS queue. 

Here are some examples of IAM policies for Amazon SQS:

1. Allow a user to create queues and perform all actions on them:

{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "sqs:*",      "Resource": "arn:aws:sqs:*:123456789012:alice_queue_*"    }  ] }

In this example, the user can access all Amazon SQS actions, but only with queues whose names are pr....

      2. Allow developers to write messages to a shared queue:

{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "sqs:SendMessage",      "Resource": "arn:aws:sqs:*:123456789012:MyCompanyQueue"    }  ] }

In this example, a group of developers can use the Amazon SQS SendMessage action, but only with the ....

    3. Allow managers to get the general size of queues:

{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "sqs:GetQueueAttributes",      "Resource": "*"    }  ] }

In this example, a group of managers can use the Amazon SQS GetQueueAttributes action with all of th....

Remember to replace 123456789012 with your actual AWS account ID, and alice_queue_ or MyCompanyQueue with your actual SQS queue name.

[Kaniz]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!