โ12-05-2023 11:49 AM
Hi,
I have a Structured Stream which reads data from my silver layer and creates a gold layer using foreachBatch. The stream has been working fine, but now I have change where there are deletions to the schema and some of the columns from the silver layer are deleted. When I try to run the stream I am facing a schema mismatch error. Is it possible for me to handle such schema changes(especially deletions) without changing the checkpoint location?
eg:
Error:
Please try restarting the query. If this issue repeats across query restarts without making progress, you have made an incompatible schema change and need to start your query from scratch using a new checkpoint directory.
โ12-05-2023 10:27 PM
Hi @Dp15, Handling schema changes in Structured Streaming can be tricky, especially when dealing with deletions or other modifications.
Letโs explore some strategies to address this issue:
Schema Evolution:
Checkpointing and Schema Changes:
Workaround:
Considerations:
In summary, while changing the checkpoint location, consider migrating existing data and updating downstream consumers to minimize disruptions caused by schema changes. Remember to test thoroughly to ensure correctness. ๐
โ12-05-2023 10:27 PM
Hi @Dp15, Handling schema changes in Structured Streaming can be tricky, especially when dealing with deletions or other modifications.
Letโs explore some strategies to address this issue:
Schema Evolution:
Checkpointing and Schema Changes:
Workaround:
Considerations:
In summary, while changing the checkpoint location, consider migrating existing data and updating downstream consumers to minimize disruptions caused by schema changes. Remember to test thoroughly to ensure correctness. ๐
โ12-06-2023 07:01 AM
@Kaniz_Fatma Thank you so much for a detailed explanation
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!