Unable to Upcast DECIMAL Field in Autoloader

[yit]

I’m using Autoloader to read Parquet files and write them to a Delta table. I want to enforce a schema in which Column1 is defined as DECIMAL(10,2). However, in the Parquet files being ingested, Column1 is defined as DECIMAL(8,2).

When Autoloader reads the file, it considers this a schema mismatch and moves the column into _rescued_data.

I’ve tried specifying the schema explicitly using .schema() as well as using schemaHints, but neither approach resolves the issue — Autoloader still treats the field as incompatible and rescues it.

I've also tried with changing the values for "cloudFiles.inferColumnTypes" and "cloudFiles.schemaEvolutionMode", but nothing works.

Is it possible to upcast DECIMAL with autoloader? If so, how?

[kerem]

Hi Yit,

To potentially simplify your issue, why not read this column as String in your stream and then cast it to DECIMAL(10, 2) afterwards? That should eliminate the rescue behaviour. 

Kerem Durak