Autoloader File Notifications mode S3 Access Denied error

[deng_dev]

Hi everyone!
We are reading json files from cross-account S3 bucket using Autoloader and decided to switch from directory listing mode to files notification mode.
We have set up all permissions mentioned here in our IAM role. But now the pipeline is failing with error:

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 9BFMJKWPPC2Z5QCS; S3 Extended Request ID: aFXXeekrRMcfxfxDWrZ1FT821bWM4H8f+/a/p7hjgNDhZvAWo7+0iHLhnL11Zvy1Laa0dgPYLTI=; Proxy: null), S3 Extended Request ID: aFXXeekrRMcfxfxDWrZ1FT821bWM4H8f+/a/p7hjgNDhZvAWo7+0iHLhnL11Zvy1Laa0dgPYLTI=

Is it not possible to use file notifications mode when reading data from cross-account S3 bucket? Or could we missed something in setting up the permissions?

Thank you!

[drewipson]

You need to be sure you have an instance profile configured with PassRole permissions so that it can assume the cross account role to access the bucket and file notification resources. I found this technical blog helpful: https://community.databricks.com/t5/technical-blog/how-to-use-databricks-autoloader-across-aws-accou...