When there is a schema change while reading and writing to a stream, will the schema changes be automatically handled by spark
or do we need to include the option(mergeschema=True)?
Eg:
df.writeStream
.option("mergeSchema", "true")
.format("delta")
.outputMode("append")
.option("path","/data/")
.option("checkpointLocation","/checkpoint/")
.start()
.awaitTermination()
Including the option(mergeschema=True) still throws the error :
ERROR: A schema mismatch detected when writing to the Delta table
To enable schema migration, please set:
'.option("mergeSchema", "true")'.
Do any additional option /changes needs to be done to the above query? Could you please advise to resolve this issue?