How to use change data feed when delta table schema changes between delta table versions?
I tried to read change data feed in parts (in code snippet I read version 1372, because 1371 and 1373 schema versions are different), but getting error
UnsupportedOperationException: [DELTA_CHANGE_DATA_FEED_INCOMPATIBLE_DATA_SCHEMA] Retrieving table changes between version 1372 and 1372 failed because of an incompatible data schema.
Your read schema is {"type":"struct","fields":[{"name":..."metadata":{}}]} at version 1375, but we found an incompatible data schema at version 1372.
Code snippet:
df_test = (
spark.read.format('delta')
.option("readChangeFeed", "true")
.option("startingVersion", 1372)
.option("endingVersion", 1372)
.load(path)
)
df_test.display()
So does that mean that change data feed is using the last (or the first) feeding version schema? Can this be changed? Streaming is not option (in streaming you can add schema change tracking).
Is there any trick to get this working?
