- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-11-2025 06:40 AM
We are currently implementing Autoloader for JSON files with nested struct fields. The goal is to detect the fields as structs, and to have schema evolution.
The schema evolution mode is set to addNewColumns, and inferColumnTypes option is set to true to detect the real types of the fields instead of making them all strings.
One of the fields is deeply nested struct. There are some empty files in the folder as well. The problem is that Autoloader infers the field as string.
What could be the issue?
I cannot use schemaHints to define the field as struct, because not even one field exists in all files, while schemaHints expects to define at least one field for struct types.