Louis_Frolio
Databricks Employee
Databricks Employee

Hey @MRTN , I did some digging and here is what I found: 

This pattern almost always means “checkpoint parquet schema that Photon doesn’t like,” not a core Delta protocol issue.

What you’re seeing lines up cleanly:

  • All-purpose cluster works because it’s using the standard Spark Parquet reader.

  • Serverless (SQL + PySpark) fails because it’s using Photon’s Parquet reader, which is stricter about schema mismatches and type conversions, and it throws FAILED_CONVERT_PARQUET_COLUMN when it encounters something inconsistent.

  • The error points directly at _delta_log/00000000000000000100.checkpoint.parquet, which means this is not your data files; it’s the checkpoint parquet that delta-rs wrote.

Given the table is maintained by delta-rs, the most likely root causes are:

  • A column type mismatch inside the checkpoint (for example, mixing INT32 and INT64, or DOUBLE and DECIMAL for the same logical field, or using a type Photon doesn’t support for that checkpoint column). Photon is less forgiving than the non-Photon reader.

  • Or a checkpoint encoding/format that your serverless runtime (Photon build) doesn’t fully support, while your all-purpose runtime happens to tolerate it.

 

Hope this helps, Louis.