Brahmareddy
Esteemed Contributor II

Hi @RoelofvS,

I have gone through your response and here is my suggestion below.

Make sure to allow for a slight delay or checkpoint refresh after each schema change to ensure Auto Loader registers updates fully. Given that renaming 0 to 1 prompted schema evolution once, try incremental versioning by creating successive files (0, 1, etc.) after each change. Additionally, consider restarting the stream whenever you make schema modifications, as this can help Auto Loader refresh the schema properly. Setting an explicit schema with .schema() for the initial load may also stabilize the evolution process by providing a structured base. For more insights, enable detailed logging to trace schema evolution steps and check for any timing or metadata conflicts.

Hope this helps!

Good day.

Regards,

Brahma