Different configurations for same Databricks Runtime version

[yoniau]

Hi all,

On my DBR installations, s3a scheme is mapped to shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem. On my customer's DBR installations it is mapped to com.databricks.s3a.S3AFileSystem.

We both use the same DBR runtime, and none of us has configured anything to override this setting.

What is the cause for this difference? And how can I make sure I'm using the right filesystem? How can I make sure in the future no third file system appears and breaks my code again?

[Hubert-Dudek]

@Yoni Au​ , Databricks Runtime 7.3 LTS and above use the new connector com.databricks.s3a.S3AFileSystem. Are you using 7.3?

Anyway, please verify spark config on both installations (via Cluster -> Spark UI -> Environment) what is there regarding S3AFileSystem? and then set common values for both (via Cluster -> Configuration -> Advanced options)

[Prabakar · Accepted solution]

@Yoni Au​ , If both of you are using the same DBR version, then you should not find any difference. As @Hubert Dudek​ mentioned, there might be some spark configuration change made on one of the clusters. Also, it's worth checking for any cluster scope or global init script.

[Kaniz]

Hi @Yoni Au​ , Just a friendly follow-up. Do you still need help, or do @Hubert Dudek (Customer)​ and @Prabakar Ammeappin​ 's responses help you find the solution? Please let us know.