It's still hard to figure out exactly what's wrong, but my guess is the explode is creating a huge dataframe that's not able to fit into memory. It largely depends on how many rows you have and the size of the struct. if you have 100 rows and the struct is length/size 100 then you get 100x100 rows.