Reading multi-dimensional json files

AndriusVitkausk — Wed, 07 Dec 2022 13:31:55 GMT

So I've been having some issues reading a json file that's been provided to the business with another nesting layer, so instead of a json being an:

'array of objects' -> [ {} ,{} ,{} ]
It's an 'array of arrays of objects' -> [ [ {}, {} ,{} ], [ {} ,{} ,{} ] ]

While the first is alright to read with the multiline option with spark, the second case simply comes with the correct column schema, thought every columns is just a null value (actual file content looks good)

I've so far tried to create a custom struct schema to deal with the extra layer, but not had any luck to get it to work. Just returns nulls.

Is there something obvious that i'm missing?

Re: Reading multi-dimensional json files

ashish1 — Mon, 30 Jan 2023 21:20:09 GMT

You can use the explode function to flatten the array to rows, can you post a simple example of your data?

topic Reading multi-dimensional json files in Data Engineering

Reading multi-dimensional json files

Re: Reading multi-dimensional json files