Hello​ @Ayesha Rahmatali​ Could you please let me know which DBR version you are using?

The below error can occur if new partition columns are being inferred from your files which cause the issue.

If thats the case, in order to resolve the issue, please provide all partition columns in your schema or provide a list of partition columns which you would like to extract values for by using: .option("cloudFiles.partitionColumns", "<comma-separated-list|empty-string>". AutoLoader infers the partition columns as empty. Use cloudFiles.partitionColumns to explicitly parse columns from the directory structure.

For more input, kindly refer to the below document.

Reference: https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-schema.html