- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-13-2025 07:35 PM
Absolutely ChristianRRL, when your nested schema isn't fully known or consistent, schemaHints can only take you so far before you need a workaround. A flexible pattern in these cases is to first ingest the raw JSON using permissive mode (with schema inference or minimal hints), and then flatten and transform the data using from_json or explode + selectExpr logic in a second stage, once you have more control. This two-stage approach gives you better resilience against edge cases, and also allows you to isolate logic for future schema evolution. And yes, here's hoping Databricks @Sujitha improves schemaHints to support wildcards and more advanced nested patterns—would definitely make streaming and ingestion pipelines smoother.