Reading multi-dimensional json files

AndriusVitkausk
New Contributor III

So I've been having some issues reading a json file that's been provided to the business with another nesting layer, so instead of a json being an:

  • 'array of objects' -> [ {} ,{} ,{} ]
  • It's an 'array of arrays of objects' -> [ [ {}, {} ,{} ], [ {} ,{} ,{} ] ]

While the first is alright to read with the multiline option with spark, the second case simply comes with the correct column schema, thought every columns is just a null value (actual file content looks good)

I've so far tried to create a custom struct schema to deal with the extra layer, but not had any luck to get it to work. Just returns nulls.

Is there something obvious that i'm missing?