- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-06-2026 01:29 PM
Thanks for the response. As a final verification, my understanding is that Autoloader schema inference and schema evolution only tracks when colums are added in our schema, which then get get stored as the new schema version. In all the rest cases - column drops, type mismatches (such as float to an int col) the schema remains the same and will get rescued or ignored as per our rescue column settings. Therefore even if our target delta table has type widening enabled - since our schema doesn’t widen itself our target delta table will not get widened via autoloader unless we manually update our schema column data type to allow accurate parsing.
1) Could you tell me if there is any configuration/option/parameter/setting to override this behavior and force schema data type widening to columns?
2) Is a same behavior seen even if we are working with Delta Live Tables?
3) And as sort of a inspiration is there any approach you would suggest to handle such schema changes when we are working with upstream data (100s of columns) that keeps on changing so providing an explicit schema or schema hints is counterproductive, and our initial inference may result in too narrow types causing mismatches in the future.
Apologies for the long questions, I am trying to understand Autoloader functions and capabilities to handle such edge cases when upstream is not stable and clean.
Thanks,
Parth