TIMESTAMP(NANOS,false) error

staskh · ‎02-20-2025

Hi,

I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks). Unfortunately, due to security configuration, I do not have the ability to read it with pandas or similar - only spark.read.parquet(). Any ideas what to do?

Regards

Stas

Alberto_Umana · ‎02-20-2025

Hi @staskh

That error happens because data type is not supported natively. You can try with below spark setting:

spark.conf.set(“spark.sql.legacy.parquet.nanosAsLong”, “true”)

View solution in original post