parsing json string value column into dataframe structure

dipali_globant
Databricks Partner

Hi All,

I have to read kafka payload which has value column with json string. But the format of the json is as below.

{ "data": [ { "p_al4": "N/A", "p_a5": "N/A", "p_ad": "OA003", "p_aName": "Abc", "p_aFlag": true ,....(dynamic)} ] }

In data key it can have dynamic number of key value pairs.

here "p_" is a column and respective value is given.
Please suggest how to create this payload into dataframe structure.
like below:

+-----+-----+-----+-------+------+------------------+
|p_al4 | p_a5 | p_ad | p_aName | p_aFlag|
+-----+-----+-----+-------+------+----------------+
|N/A | N/A | OU13| Admin | true |
+-----+-----+-----+-------+------+--------------