Hi,
written in pyspark.

databricks autoloader job with retry didn't merge/update the schema.

spark.readStream.format("cloudFiles")

.option("cloudFiles.format", "parquet")

.option("cloudFiles.schemaLocation", checkpoint_path)

.option("cloudFiles.includeExistingFiles", "false"

.load(source_path)

.writeStream

.queryName("write stream query")

.option("checkpointLocation", checkpoint_path)

.trigger(availableNow=True)

.forEachBatch(batch_operation)

.option("mergeSchema", True)

.start()

.awaitTermination()

Error: 

Error while reading file s3://path
Caused by: UnknownFieldException: [UNKNOWN_FILED_EXCEPTION.NEW_FILEDS_IN_FILE] Encountered unknown fields during parsing: filed type, which can be fixed by an automatic retry: false

tried running couple of times. set retry = 2 for job and task as well.

please can you help?