Parquet file for delta streaming live table with pipeline

ClarkElliott — Tue, 18 Mar 2025 21:26:50 GMT

I am having an issue with parquet files:

I'm getting Illegal Parquet type: INT64 (TIMESTAMP(NANOS,false)) error while trying to read a parquet file (generated outside of DataBricks).

I am using a Delta streaming live table with a pipeline. If I remove the one file the pipeline works fine. I can use Pandas in Python can open this file just fine. Any ideas on how to address this with a pipeline based on a notebook with a single cell create streaming live table .... I've seen I can add this: spark.conf.set("spark.sql.legacy.parquet.nanosAsLong", "true") If using spark directly. Not sure on how to affect the abstraction of this loading using the delta streaming live table.

Any help appreciated

Re: Parquet file for delta streaming live table with pipeline

Saritha_S — Wed, 11 Jun 2025 17:39:34 GMT

Hi @ClarkElliott

Good day!!

Cause

Databricks Runtime versions 11.3 LTS and above do not support the TIMESTAMP_NANOS type in open source Apache Spark and Databricks Runtime. If a Parquet file contains fields with the TIMESTAMP_NANOS type, attempts to read it will fail with an Illegal Parquet Type exception. As a result, schema inference will also fail, since Spark cannot interpret the unsupported timestamp type.

To restore the behavior before Spark 3.2, you can set spark.sql.legacy.parquet.nanosAsLong to true.
Reference: https://spark.apache.org/docs/4.0.0/sql-migration-guide.html#upgrading-from-spark-sql-31-to-32:~:text=To%20restore%20the%20behavior%20before%20Spark%203.2%2C%20you%20can%20set%20spark.sql.legacy.parquet.nanosAsLong%20to%20true.

You can add the below configuration to the DLT pipeline settings.

spark.sql.legacy.parquet.nanosAsLong true

Kindly let me know if you have any questions on this.

topic Parquet file for delta streaming live table with pipeline in Data Engineering

Parquet file for delta streaming live table with pipeline

Re: Parquet file for delta streaming live table with pipeline

Cause