topic Re: an autoloader in file notification mode to get files from S3 on AWS -Error in Get Started Discussions

an autoloader in file notification mode to get files from S3 on AWS -Error

Chris_Konsur — Tue, 06 Feb 2024 14:19:27 GMT

I configured an autoloader in file notification mode to get files from S3 on AWS.

spark.readStream\
.format("cloudFiles")\
.option("cloudFiles.format", "json")\
.option("cloudFiles.inferColumnTypes", "true")\
.option("cloudFiles.schemaLocation", "dbfs:/auto-loader/schemas/")\
.option("cloudFiles.useNotifications", "true")\
.option("includeExistingFiles", "true")\
.option("multiLine", "true")\
.option("inferSchema", "true")\
.load("s3://orcus-rave-bucket/temp/cludcad_incident3")\
.writeStream\
.option("checkpointLocation", "dbfs:/auto-loader/checkpoint04/")\
.trigger(availableNow=True)\
.table("al_table3")

I configured IAM rule based on the document URL: https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/file-notification-mode#--required-permissions-for-configuring-file-notification-for-aws-s3

Re: an autoloader in file notification mode to get files from S3 on AWS -Error

Chris_Konsur — Tue, 06 Feb 2024 14:21:22 GMT

Here is an error:

I get the following error:

com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [BasicAWSCredentialsProvider: Access key or secret key is null, com.amazonaws.auth.InstanceProfileCredentialsProvider@5ab84d23: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]

Re: an autoloader in file notification mode to get files from S3 on AWS -Error

Selz — Fri, 08 Mar 2024 14:20:09 GMT

I have the same error after adding the IAM permissions noted in the file notification mode documentation. Were you able to find a solution?

Re: an autoloader in file notification mode to get files from S3 on AWS -Error

Selz — Wed, 20 Mar 2024 15:17:58 GMT

In case anyone else stumbles across this, I was able to fix my issue by setting up an instance profile with the file notification permissions and attaching the instance profile to the job cluster. It wasn't clear from the documentation that the file notification permissions can't be set up with a role and job using storage credentials. This article helped: https://medium.com/@mattwinmill88/deploying-a-databricks-aws-end-to-end-pipeline-using-terraform-921c6b4a36a7

Re: an autoloader in file notification mode to get files from S3 on AWS -Error

aliehs0510 — Thu, 21 Nov 2024 07:05:40 GMT

Hi @Selz ,

I currently have the same error when running autoloader on file notification mode. I have done the following steps:
1. setup instance profile with file notification permissions

2. added the instance profile on databricks workspace , settings->security-> instance profiles
3. configured the job compute policy to add the config

"aws_attributes.instance_profile_arn": { "type": "allowlist", "values": [ "arn:aws:iam::<account_id>:instance-profile/<my instance profile role>" ], "isOptional": true },

however I'm still getting the same error. wondering if I'm I did something wrong or missed a step. I appreciate your guidance on this.