Hi All, I am trying to read streams directly from AWS S3. I set the instance profile , but when i run the workflow it fails with below error
"No AWS Credentials provided by TemporaryAWSCredentialsProvider : shaded.databricks.org.apache.hadoop.fs.s3a.CredentialInitializationException: Access key, secret key or session token is unset: "
I added below to my cluster
fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
fs.s3a.access.key <AccessKeyId>
fs.s3a.secret.key <SecretAccessKey>
It still fails with same error. Could some one please help me how to pass these for DLT workflows