04-26-2022 05:15 AM
I have a task to transform a dataframe.
The task is to collect all the columns in a row and embed it into a JSON string as a column.
Source DF:
Target DF:
04-27-2022 12:14 AM
I was able to do this by converting df to rdd and then by applying map function to it.
rdd_1 = df.rdd.map(lambda row: (row['ID'], row.asDict() ) ) rdd_2_df = rdd_1.toDF(['ID', 'Data'])
View solution in original post
05-13-2022 06:12 AM
Hi @Aman Sehgal, Thank you for providing the solution here. I'm marking your answer as the best.
never-displayed
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.