I am having an issue with parquet files:  

 

I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks). 

 

I am using a Delta streaming live table with a pipeline.  If I remove the one file the pipeline works fine.  I can use Pandas in Python can open this file just fine.  Any ideas on how to address this with a pipeline based on a notebook with a single cell  create streaming live table ....    I've seen I can add this: spark.conf.set("spark.sql.legacy.parquet.nanosAsLong", "true")  If using spark directly.  Not sure on how to affect the abstraction of this loading using the delta streaming live table.

 

Any help appreciated