@BS_THE_ANALYST @szymon_dybczak thank you both for your replies.

The solution is already implemented in Auto Loader — it does exactly what I needed, even for nested fields.

The behavior that confused me was that when the evolution mode is set to addNewColumns and a file with new schema fields is processed, an error is initially raised for the new fields. At that same moment, however, the new merged schema is saved to the schema location. If we then reprocess the file (by running the stream again), it is processed correctly and schema evolution succeeds. The new fields are added and populated with null values for existing records.