Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2024 02:33 AM - edited 06-12-2024 02:35 AM
This is a thoughtful consideration, but have you considered using
.option("mergeSchema", "true")
when writing?
Do keep in mind that this will affect the target table and possible downstream consumers. Ideally you want to have strict a schema contract with your data suppliers to avoid these issues. You can also consider dynamically creating the model.json based on the file headers you're receiving.