I was able to do this by converting df to rdd and then by applying map function to it.

rdd_1 = df.rdd.map(lambda row: 
                             (row['ID'],
                             row.asDict()
                             )
              )
 
rdd_2_df = rdd_1.toDF(['ID', 'Data'])