Can schemaHints dynamically handle nested json structures?

ChristianRRL
Honored Contributor II

Hi there, as I'm learning more about schemaHints, it seems like an incredibly useful way to unpack some of my json data. However, I've hit what is either a limitation of schemaHints or of my understanding of how to use it properly.

Below I have an example of a somewhat standard read_files query. However, rather than applying schemaHints to a single nested structure, I would like to apply a schemaHints pattern to multiple nested fields inside *all* structs inside a given sub_data field (e.g. everything inside elementData.element.data.*).

ChristianRRL_0-1755017647152.png

However, when I try using the `*` wildcard, I quickly get an error:

ChristianRRL_2-1755018810157.png

The only way I've seen so far to get close to (but not quite) where I'm wanting to get, is I can apply a double map<string,string> on the data field, like so:

ChristianRRL_3-1755019651215.png

But the challenge is I want to map two separate sub-fields (dataPoint & values).

If I try to triple map, I lose either the dataPoint field or the values field (depending on how I map it):

e.g. Mapping dataPoint properly (but values is lost)

ChristianRRL_6-1755020076906.png

e.g. Mapping values properly (but dataPoint is lost)

ChristianRRL_5-1755019866487.png

Any feedback/input on this would be much appreciated!

+Side-note: Props to @genevive_mdonça for your post Handling Complex Nested JSON in Databricks Using s... - Databricks Community - 116210. This was a great springboard!