cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

why the userIdentity is anonymous?

rt-slowth
Contributor

Do you know why the userIdentity is anonymous in AWS Cloudtail's logs even though I have specified an instance profile?

6 REPLIES 6

CharlesReily
New Contributor III

If you're using AssumeRole to switch roles, make sure that the assumed role session is being used correctly. The Security Token Service (STS) is responsible for issuing temporary security credentials when assuming roles. Ensure that your EC2 instances have been assigned the correct IAM roles with the necessary permissions. AWS CloudTrail logs typically capture the identity of the caller making the API request. If your EC2 instance has the correct IAM role associated with it, CloudTrail should log the identity appropriately.

@Kaniz_Fatma @CharlesReily 

Directory listing mode is supported by default. File notification mode is only supported on single user clusters.

 
I was trying to use File Notification Mode in DLT and encountered the above problem...

I tried with "Single User" cluster, but still getting error "org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = 55e0cb16-5e12-444d-a132-a24b999e2e4a, runId = 3ae637bb-847a-472b-a133-64b58ccb35cb] terminated with exception: User: anonymous is not authorized to perform: sqs:receivemessage on resource: arn:aws:sqs:us-east-1:4".

Used "data_security_mode":"SINGLE_USER" in Cluste policy. Am I missing anything ?

@Babu_Krishnan 

As far as I know, pipelines created with Shared Cluster and Delta Live Table are not in File notification mode.
Since Delta Live Table is a Shared Cluster by default.
Before that, how is your AWS IAM role set up?
If you can share your cluster configuration and the code that performs the readStream, I'll see what I can do.

@rt-slowth ,  Thanks for the reply. IAM role should be fine (with all the required permissions) since it was perfectly working with DLT without UC. We are seeing this failure when we are migrating. the existing DLT to Unity Catalog. FYI , I am able run the pipeline with "Direcltory listing" mode, I see this SQS permission error only when we use the file notification mode.

Babu_Krishnan_0-1714616743848.png

Babu_Krishnan_1-1714616817428.png

 

 

Kaniz_Fatma
Community Manager
Community Manager

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 
 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group