I am having an issue with parquet files:
I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks).
I am using a Delta streaming live table with a pipeline. If I remove the one file the pipeline works fine. I can use Pandas in Python can open this file just fine. Any ideas on how to address this with a pipeline based on a notebook with a single cell create streaming live table .... I've seen I can add this: spark.conf.set("spark.sql.legacy.parquet.nanosAsLong", "true") If using spark directly. Not sure on how to affect the abstraction of this loading using the delta streaming live table.
Any help appreciated
