AutoLoader Type Widening

parth_db · ‎01-05-2026

I have a few doubts regarding AutoLoader behavior and capabilities. Please check and correct wherever my assumptions or understanding are incorrect, much appreciated. Below is my specific code

Example scenario:
Target Managed Delta Table (Type Widening enabled) has a field - 'Quantity' , Type - int.

My code:

df = (

spark.readStream

.format("cloudFiles")

.option("cloudFiles.format", "csv")

.option("cloudFiles.schemaLocation", schema_location)

# Schema Inference
.option("cloudFiles.inferColumnTypes", "true")

.option("cloudFiles.schemaEvolutionMode", "addNewColumns")

.option("cloudFiles.rescuedDataColumn", "_rescued_data")

.option("cloudFiles.allowOverwrites", "true")

.option("header", "true")

.load(source_dir)

.withColumn("_rescued_data", col("_rescued_data").cast("string"))

)

def upsert_batch(microBatchDF, batchId: int😞

# If target table doesn't exist yet, bootstrap it from inferred schema

if not table_exists_uc(target_table):

(microBatchDF.write

.format("delta")

.mode("overwrite")

.saveAsTable(target_table)

)

return

delta_t = DeltaTable.forName(spark, target_table)

(delta_t.alias("t")

.merge(microBatchDF.alias("s"), "t.order_id = s.order_id")

.withSchemaEvolution()

.whenMatchedUpdateAll()

.whenNotMatchedInsertAll()

.execute()

)

query = (

df.writeStream

.foreachBatch(upsert_batch)

.option("checkpointLocation", checkpoint_location)

.trigger(availableNow=True)

.start()

)

Let's say on our first run we ingest a file which has all rows with quantity field as int, so my assumption our stored schema 0 should have quantity inferred as an int.
On our next run we load a file which has float values in the quantity column. So tell me, in this case will our target delta table automatically widen to accomodate this new file data which is a float? Before our writestream happens in the readstream portion itself my understanding is that since the latest schema it has mentions quantity with type int, the readstream will try to parse the float values against the int type and will not be able to do so these will end in rescued data and hence the target table quantity field will never see these float values. Please confirm this behavior.
Also as an additional question, does Autoloader ever parse floats to int (Ex : 4.5 -> 4 OR 2.0 -> 2) and if not, why?

Thanks,
Parth