Schema Evolution Issue in Streaming
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-03-2022 03:17 AM
When there is a schema change while reading and writing to a stream, will the schema changes be automatically handled by spark
or do we need to include the option(mergeschema=True)?
Eg:
df.writeStream
.option("mergeSchema", "true")
.format("delta")
.outputMode("append")
.option("path","/data/")
.option("checkpointLocation","/checkpoint/")
.start()
.awaitTermination()
Including the option(mergeschema=True) still throws the error :
ERROR: A schema mismatch detected when writing to the Delta table
To enable schema migration, please set:
'.option("mergeSchema", "true")'.
Do any additional option /changes needs to be done to the above query? Could you please advise to resolve this issue?
- Labels:
-
Schema Evolution Issue