TIMESTAMP(NANOS,false) error

[staskh]

Hi,

I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks). Unfortunately, due to security configuration, I do not have the ability to read it with pandas or similar - only spark.read.parquet(). Any ideas what to do?

Regards

Stas 

[Alberto_Umana · Accepted solution]

Hi @staskh

That error happens because data type is not supported natively. You can try with below spark setting:

spark.conf.set(“spark.sql.legacy.parquet.nanosAsLong”, “true”)