Hi @robertomatus ,

You're right—it would be much better if we didn’t have to rely on workarounds. The reason AutoLoader infers schema differently from spark.read.json() is that it's optimized for streaming large-scale data efficiently. Unlike spark.read.json(), which scans all files, AutoLoader samples data to infer schema faster and supports incremental schema evolution for handling new columns over time.

If you want a more reliable approach, consider defining the schema manually and passing it to AutoLoader instead of relying on inference. Another option is to use spark.read.json() on a small sample once, extract the schema, and then provide it to AutoLoader. You can also enable schema evolution using .option("cloudFiles.schemaEvolutionMode", "rescue") to handle unexpected changes dynamically.

While it would be great if AutoLoader handled this seamlessly, these steps can help make schema inference more predictable and reduce inconsistencies.

Hoping you have a good day.

Regards,

Brahma