Hi
I'm importing a large collection of json files, the problem is that they are not what I would expect a well-formatted json file to be (although probably still valid), each file consists of only a single record that looks something like this (this is just an abstraction)
[{"name":"myName","Surname":"MySurname"},[{"address":"1","Type":"Home"},{"address":"2","Type":"Home"}],[{"Tel":"1"},{"Tel":"2"}]]
I would ideally like to import it using the standard json read option bus can't figure out how to structure the schema.
My first approach involved creating a UDF that imported the record as a string and returned a new properly formatted object
def structure(object):
structure = {}
structure["name"] = object[0]["name"]
structure["Surname"] = object[0]["Surname"]
structure["addresses"] = []
structure["telephones"] = []
for address in object[1]:
structure["addresses"].append({"address": address["address"],"Type": address["Type"]})
for telephones in object[2]:
structure["telephones"].append({"Tel": telephones["Tel"]})
return structure
This works but it will be slower and less intuitive.
using a schema like this "works" but the elements in the ID'd come back as null
mySchema = StructType([
StructField("Name", StringType(), True),
StructField("Surname", StringType(), True),
StrutType(StructField("IDs",
StructType([StructField("ID", StringType(), True)])
)
)
])
