Databricks Community

AmanSehgal · ‎04-26-2022

I have a task to transform a dataframe.

The task is to collect all the columns in a row and embed it into a JSON string as a column.

Source DF:

Target DF:

AmanSehgal · ‎04-27-2022

I was able to do this by converting df to rdd and then by applying map function to it.

rdd_1 = df.rdd.map(lambda row: 
                             (row['ID'],
                             row.asDict()
                             )
              )
 
rdd_2_df = rdd_1.toDF(['ID', 'Data'])

View solution in original post

AmanSehgal · ‎04-27-2022

I was able to do this by converting df to rdd and then by applying map function to it.

rdd_1 = df.rdd.map(lambda row: 
                             (row['ID'],
                             row.asDict()
                             )
              )
 
rdd_2_df = rdd_1.toDF(['ID', 'Data'])