How to handle Blank values in Array of struct elements in pyspark
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-07-2020 11:14 PM
Hello All,
We have a data in a column in pyspark dataframe having array of struct typehaving multiple nested fields present.if the value is not blank it will savethe data in the same array of struct type in spark delta table.please advise on the below case:if the same column coming as blank ,it is treated as array<string> in thedataframe.f the same column coming as array of struct elements it is fine.Could you please advise how do we handle this scenario in pyspark to avoidthe error "Failed to merge incompatible data types StructType(StructField(ac...................... and StringTypehow do we handle array of struct type and array<string> in the same columnThanks
Labels:
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-15-2020 12:05 PM
Hi @Anbazhagan anbutech17,
Can you please try as in below answers,https://stackoverflow.com/questions/56942683/how-to-add-null-columns-to-complex-array-struct-in-spar...