https://community.databricks.com/t5/data-engineering/unable-to-upcast-decimal-field-in-autoloader/m-p/127506#M47990 <P>I’m using Autoloader to read Parquet files and write them to a Delta table. I want to enforce a schema in which Column1 is defined as DECIMAL(10,2). However, in the Parquet files being ingested, Column1 is defined as DECIMAL(8,2).</P><P>When Autoloader reads the file, it considers this a schema mismatch and moves the column into _rescued_data.</P><P>I’ve tried specifying the schema explicitly using .schema() as well as using schemaHints, but neither approach resolves the issue — Autoloader still treats the field as incompatible and rescues it.</P><P>I've also tried with changing the values for&nbsp;<SPAN>"cloudFiles.inferColumnTypes" and</SPAN><SPAN>&nbsp;</SPAN><SPAN>"cloudFiles.schemaEvolutionMode", but nothing works.</SPAN></P><P><SPAN>Is it possible to upcast DECIMAL with autoloader? If so, how?</SPAN></P> Tue, 05 Aug 2025 21:19:13 GMT yit 2025-08-05T21:19:13Z https://community.databricks.com/t5/data-engineering/unable-to-upcast-decimal-field-in-autoloader/m-p/127506#M47990 <P>I’m using Autoloader to read Parquet files and write them to a Delta table. I want to enforce a schema in which Column1 is defined as DECIMAL(10,2). However, in the Parquet files being ingested, Column1 is defined as DECIMAL(8,2).</P><P>When Autoloader reads the file, it considers this a schema mismatch and moves the column into _rescued_data.</P><P>I’ve tried specifying the schema explicitly using .schema() as well as using schemaHints, but neither approach resolves the issue — Autoloader still treats the field as incompatible and rescues it.</P><P>I've also tried with changing the values for&nbsp;<SPAN>"cloudFiles.inferColumnTypes" and</SPAN><SPAN>&nbsp;</SPAN><SPAN>"cloudFiles.schemaEvolutionMode", but nothing works.</SPAN></P><P><SPAN>Is it possible to upcast DECIMAL with autoloader? If so, how?</SPAN></P> Tue, 05 Aug 2025 21:19:13 GMT https://community.databricks.com/t5/data-engineering/unable-to-upcast-decimal-field-in-autoloader/m-p/127506#M47990 yit 2025-08-05T21:19:13Z https://community.databricks.com/t5/data-engineering/unable-to-upcast-decimal-field-in-autoloader/m-p/127514#M47994 <P>Hi Yit,</P><P>To potentially simplify your issue, why not read this column as String in your stream and then cast it to DECIMAL(10, 2) afterwards? That should eliminate the rescue behaviour.&nbsp;<BR /><BR /></P><P>Kerem Durak</P> Tue, 05 Aug 2025 23:02:24 GMT https://community.databricks.com/t5/data-engineering/unable-to-upcast-decimal-field-in-autoloader/m-p/127514#M47994 kerem 2025-08-05T23:02:24Z