how to dynamically explode array type column in pyspark or scala

SatheeshSathees — Wed, 19 Aug 2020 18:31:33 GMT

HI,

i have a parquet file with complex column types with nested structs and arrays.

I am using the scrpit from below link to flatten my parquet file.

https://docs.microsoft.com/en-us/azure/synapse-analytics/how-to-analyze-complex-schema

I am able to flatten schema using script in STEP 1 and STEP 2 successfully. But in the above link, for STEP 3 the script uses hardcoded column names to flatten arrays. But in my case i have multiple columns of array type that need to be transformed so i cant use this method.

Is there any way to dynamically transform all the array type columns without hardcoding because in future the columns may change in my case. Something like check if a column is of array type and explode it dynamically and repeat for all columns of arrays.

Please advise

topic how to dynamically explode array type column in pyspark or scala in Data Engineering

how to dynamically explode array type column in pyspark or scala

Re: how to dynamically explode array type column in pyspark or scala