an autoloader in file notification mode to get files from S3 on AWS -Error

[Chris_Konsur]

I configured an autoloader in file notification mode to get files from S3 on AWS.

spark.readStream\
.format("cloudFiles")\
.option("cloudFiles.format", "json")\
.option("cloudFiles.inferColumnTypes", "true")\
.option("cloudFiles.schemaLocation", "dbfs:/auto-loader/schemas/")\
.option("cloudFiles.useNotifications", "true")\
.option("includeExistingFiles", "true")\
.option("multiLine", "true")\
.option("inferSchema", "true")\
.load("s3://orcus-rave-bucket/temp/cludcad_incident3")\
.writeStream\
.option("checkpointLocation", "dbfs:/auto-loader/checkpoint04/")\
.trigger(availableNow=True)\
.table("al_table3")

I configured IAM rule based on the document URL: https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/file-notification-mode#--re...

[Chris_Konsur]

Here is an error: 

I get the following error:

com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [BasicAWSCredentialsProvider: Access key or secret key is null, com.amazonaws.auth.InstanceProfileCredentialsProvider@5ab84d23: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]

[Selz]

I have the same error after adding the IAM permissions noted in the file notification mode documentation. Were you able to find a solution?

[Selz]

In case anyone else stumbles across this, I was able to fix my issue by setting up an instance profile with the file notification permissions and attaching the instance profile to the job cluster. It wasn't clear from the documentation that the file notification permissions can't be set up with a role and job using storage credentials. This article helped: https://medium.com/@mattwinmill88/deploying-a-databricks-aws-end-to-end-pipeline-using-terraform-921...