topic Re: Different configurations for same Databricks Runtime version in Data Engineering

Different configurations for same Databricks Runtime version

yoniau — Sun, 03 Apr 2022 11:05:24 GMT

Hi all,

On my DBR installations, s3a scheme is mapped to shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem. On my customer's DBR installations it is mapped to com.databricks.s3a.S3AFileSystem.

We both use the same DBR runtime, and none of us has configured anything to override this setting.

What is the cause for this difference? And how can I make sure I'm using the right filesystem? How can I make sure in the future no third file system appears and breaks my code again?

Re: Different configurations for same Databricks Runtime version

Hubert-Dudek — Mon, 04 Apr 2022 09:32:02 GMT

@Yoni Au , Databricks Runtime 7.3 LTS and above use the new connector com.databricks.s3a.S3AFileSystem. Are you using 7.3?

Anyway, please verify spark config on both installations (via Cluster -> Spark UI -> Environment) what is there regarding S3AFileSystem? and then set common values for both (via Cluster -> Configuration -> Advanced options)

Re: Different configurations for same Databricks Runtime version

Prabakar — Tue, 05 Apr 2022 07:06:24 GMT

@Yoni Au , If both of you are using the same DBR version, then you should not find any difference. As @Hubert Dudek mentioned, there might be some spark configuration change made on one of the clusters. Also, it's worth checking for any cluster scope or global init script.