topic How to merge all the columns into one column as JSON? in Data Engineering

How to merge all the columns into one column as JSON?

AmanSehgal — Tue, 26 Apr 2022 12:15:17 GMT

I have a task to transform a dataframe.

The task is to collect all the columns in a row and embed it into a JSON string as a column.

Source DF:

Target DF:

Re: How to merge all the columns into one column as JSON?

AmanSehgal — Wed, 27 Apr 2022 07:14:26 GMT

I was able to do this by converting df to rdd and then by applying map function to it.

rdd_1 = df.rdd.map(lambda row: 
                             (row['ID'],
                             row.asDict()
                             )
              )
 
rdd_2_df = rdd_1.toDF(['ID', 'Data'])