Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2022 04:53 AM
@Werner Stinckens If you look at the Schema that was shown above, it has many levels and sub-levels, like: Struct, Array. In this Schema I created is returning only null values, I don't know where I'm going wrong.
schema = StructType([
StructField('fields', StructType([
StructField('comment', StructType([
StructField("comments",ArrayType( StructField('body', StringType())),True),
])),
])),
StructField('id', StringType()),
StructField('key', StringType()),
StructField('self', StringType())
])
df = spark.createDataFrame([response],schema)
df = df.withColumn("fields", explode((("fields"))))\
.withColumn("comment", explode((("fields.comment"))))\
.withColumn("comments", explode((("comment.comments"))))