topic Re: Schema Evolution Issue in Streaming in Data Engineering

Schema Evolution Issue in Streaming

Sandy21 — Thu, 03 Nov 2022 10:17:42 GMT

When there is a schema change while reading and writing to a stream, will the schema changes be automatically handled by spark

or do we need to include the option(mergeschema=True)?

Eg:

df.writeStream

.option("mergeSchema", "true")

.format("delta")

.outputMode("append")

.option("path","/data/")

.option("checkpointLocation","/checkpoint/")

.start()

.awaitTermination()

Including the option(mergeschema=True) still throws the error :

ERROR: A schema mismatch detected when writing to the Delta table

To enable schema migration, please set:

'.option("mergeSchema", "true")'.

Do any additional option /changes needs to be done to the above query? Could you please advise to resolve this issue?

Re: Schema Evolution Issue in Streaming

Hubert-Dudek — Thu, 03 Nov 2022 10:30:26 GMT

mergeSchema doesn't support all operations. In some cases .option("overwriteSchema", "true") is needed. MergeSchema doesn't support:

Dropping a column
Changing an existing column's data type (in place)
Renaming column names that differ only by case (e.g., “Foo” and “foo”)

more on that topic here https://www.databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html

Re: Schema Evolution Issue in Streaming

Sandy21 — Fri, 04 Nov 2022 06:13:29 GMT

Thanks @Hubert Dudek . In the case of writing to a streaming table, do we need to change the checkpoint location as well in addition to adding the option("mergeSchema", "true")

if there is an addition of a new column in the incoming data?