Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-16-2025 06:22 AM
It is possible to define empty struct as column type through schema hints, but it won't do schema evolution if subfields appear in the data for that column.
Conclusion when working with JSON files and 'addNewColumns' as schema evolution mode:
- You can give partial schema through schema hints.
- If you give schema hint for some field which is of type struct, you must provide the full schema of the struct. Otherwise it will raise an error, as schema evolution does not apply for fields for which we have defined schema hints.
- If a column has data from both struct and array types, it will infer it as string - the most generic type to represent both struct and arrays.