Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-17-2023 10:03 AM
Below are topic to check:
1. Permissions mentioned here : https://docs.databricks.com/en/ingestion/auto-loader/file-notification-mode.html
2. try using SQS directly instead of letting databricks manage the SQS and SNS resources like below:
Accordingly update the SQS resource policy so that roleArn can read the messages.
.option('cloudFiles.region', 'us-xxx-2')
.option('cloudFiles.queueUrl', 'name-of-queue')
.option('cloudFiles.roleArn', 'ARN of your role')
.option('cloudFiles.roleSessionName','roleSessionName')
.option('cloudFiles.stsEndpoint','endpoint')
