topic Re: DLT workflow failing to read files from AWS S3 in Data Engineering

DLT workflow failing to read files from AWS S3

sudhanshu1 — Tue, 10 Jan 2023 12:00:49 GMT

Hi All, I am trying to read streams directly from AWS S3. I set the instance profile , but when i run the workflow it fails with below error

"No AWS Credentials provided by TemporaryAWSCredentialsProvider : shaded.databricks.org.apache.hadoop.fs.s3a.CredentialInitializationException: Access key, secret key or session token is unset: "

I added below to my cluster

fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider

fs.s3a.access.key <AccessKeyId>

fs.s3a.secret.key <SecretAccessKey>

It still fails with same error. Could some one please help me how to pass these for DLT workflows

Re: DLT workflow failing to read files from AWS S3

Vivian_Wilfred — Tue, 10 Jan 2023 14:54:32 GMT

Hi @SUDHANSHU RAJ is UC enabled on this workspace? What is the access mode set on the cluster?

Is this coming from the metastore or directly when you read from S3? Is the S3 cross-account?

Re: DLT workflow failing to read files from AWS S3

sudhanshu1 — Tue, 10 Jan 2023 15:42:44 GMT

Dear Vivian,

UC is not enabled on this workspace . I am using Instance profile set up as per databricks document .

S3 is set up for cross account and as i said , i am able to run dbutils.fs.ls("s3a://zuk-comparis-poc/")

But when i run the workflow which envoks a delta notebook ,it gives me this error.

This is a standard cluster , so i have not enabled IAM passthrough.

Am i missing something . Thanks in advance

Re: DLT workflow failing to read files from AWS S3

Vivian_Wilfred — Wed, 11 Jan 2023 12:27:40 GMT

@SUDHANSHU RAJ Can you please share the pipeline settings in JSON and also the cluster policy JSON? If this works on a standard cluster but not from a DLT pipeline, we need to verify the DLT pipeline settings for the cluster.

Re: DLT workflow failing to read files from AWS S3

sudhanshu1 — Thu, 12 Jan 2023 10:07:09 GMT

Hi Vivian,

Thanks for your help. I am happy to inform that it's working now . I think problem was in assigning proper roles and access to instance profile(in AWS) which i created for this purpose . Once i added few more rules , it started working .

Thanks again for all your help.