Hubert-Dudek
Databricks MVP

@Yoni Au​ , Databricks Runtime 7.3 LTS and above use the new connector com.databricks.s3a.S3AFileSystem. Are you using 7.3?

Anyway, please verify spark config on both installations (via Cluster -> Spark UI -> Environment) what is there regarding S3AFileSystem? and then set common values for both (via Cluster -> Configuration -> Advanced options)


My blog: https://databrickster.medium.com/