How to handle Blank values in Array of struct elements in pyspark

Anbazhagananbut — Wed, 08 Apr 2020 06:14:28 GMT

Hello All,

We have a data in a column in pyspark dataframe having array of struct type

having multiple nested fields present.if the value is not blank it will save

the data in the same array of struct type in spark delta table.

please advise on the below case:

if the same column coming as blank ,it is treated as array<string> in the

dataframe.

f the same column coming as array of struct elements it is fine.

Could you please advise how do we handle this scenario in pyspark to avoid

the error "Failed to merge incompatible data types StructType(StructField(ac

...................... and StringType

how do we handle array of struct type and array<string> in the same column

Thanks

Re: How to handle Blank values in Array of struct elements in pyspark

shyam_9 — Wed, 15 Apr 2020 19:05:23 GMT

Hi @Anbazhagan anbutech17,

Can you please try as in below answers,