NandiniN
Databricks Employee
Databricks Employee

Sorry, I am not going through the entire schema and code, but in general the error "java.lang.ArithmeticException: long overflow" typically occurs when a calculation exceeds the range that can be represented by a long data type in Java. This issue can arise in various scenarios, including data type conversions, arithmetic operations, or when handling large datasets.

You could try the below conf.

spark.sql.parquet.int96RebaseModeInWrite to CORRECTED