Schema Evolution Issue in Streaming

Sandy21
New Contributor III

When there is a schema change while reading and writing to a stream, will the schema changes be automatically handled by spark

or do we need to include the option(mergeschema=True)?

Eg:

df.writeStream

 .option("mergeSchema", "true")

 .format("delta")

 .outputMode("append")

 .option("path","/data/")

 .option("checkpointLocation","/checkpoint/")

 .start()

 .awaitTermination()

  

Including the option(mergeschema=True) still throws the error :

ERROR: A schema mismatch detected when writing to the Delta table

To enable schema migration, please set:

'.option("mergeSchema", "true")'.

Do any additional option /changes needs to be done to the above query? Could you please advise to resolve this issue?