Brahmareddy
Esteemed Contributor II

Absolutely ChristianRRL, when your nested schema isn't fully known or consistent, schemaHints can only take you so far before you need a workaround. A flexible pattern in these cases is to first ingest the raw JSON using permissive mode (with schema inference or minimal hints), and then flatten and transform the data using from_json or explode + selectExpr logic in a second stage, once you have more control. This two-stage approach gives you better resilience against edge cases, and also allows you to isolate logic for future schema evolution. And yes, here's hoping Databricks @Sujitha improves schemaHints to support wildcards and more advanced nested patterns—would definitely make streaming and ingestion pipelines smoother.

View solution in original post