How to merge all the columns into one column as JSON?

[AmanSehgal]

I have a task to transform a dataframe.

The task is to collect all the columns in a row and embed it into a JSON string as a column.

Source DF:

Target DF:

 

[AmanSehgal · Accepted solution]

I was able to do this by converting df to rdd and then by applying map function to it.

rdd_1 = df.rdd.map(lambda row: 
                             (row['ID'],
                             row.asDict()
                             )
              )
 
rdd_2_df = rdd_1.toDF(['ID', 'Data'])