ERROR yarn.ApplicationMaster: - Wrong FS s3:// expected s3a://

jay548 — Mon, 25 Jul 2022 13:40:59 GMT

We migrated from HDP to Cloudera platform 7, everything works except when we try to use databricks with redshift to load the data into a redshift table. we get the following error . ERROR yarn.ApplicationMaster: User class threw exception: java.lang.IllegalArgumentException: Wrong FS s3://path-6275110xxxxx-us-east-1-qa-edh-cdp-support//test2/c041741e-d339-4ed7-a7d3-1c20xxxxxxxx/manifest.json -ex

everywhere we are passing s3a://...... but still takes s3://

we are using below databricks - redshift dependencies

<groupId>com.amznaws</groupId>

</dependency>

<groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-aws</artifactId>

</dependency>

<groupId>com.amznaws</groupId>

</dependency>