https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#language-pyt...

you can enforce the schema or use the "cloudFiles.schemaHints"  to override the Inference.

df = spark.readStream.format("cloudFiles") \
  .option("cloudFiles.format", "csv") \
  .option("header", "true") \
  .option("rescuedDataColumn", "_rescued_data") \ # makes sure that you don't lose data
  .schema(<schema>) \ # provide a schema here for the files
  .load(<path>