how to convert each row of df to array of rows(list of rows)

[SailajaB]

Hi,

How to convert each row of dataframe to array of rows?

Here is our scenario , we need to pass each row of dataframe to one function as dict to apply the key level transformations. But as our data is very huge we can't use collect df.toJson().collect() to iterate over each row as it uses only driver's memory.

Please add your suggestions here.

Thank you

[Hubert-Dudek]

To optimize performance I would write pandas vectorized function and registered it as udf . https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.pandas_udf.html

If it is one time go you can also use basic spark function like foreach https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.foreach.html

[SailajaB]

@Hubert Dudek​ , Thank you for the reply.

We are new to ADB. And using the below code, looking for an optimized way to do it

dfJSONString = df.toJSON().collect()

stringList = []

  for row in dfJSONString:

    # ==== Unflatten the JSON string ==== #

    jsonString = unflatten(json.loads(row),dictreg[reg.upper()])

     stringList.append(json.dumps(jsonString))

Thank you

[Dan_Z · Accepted solution]

You have two options here:

1. Regular PySpark UDF, where you define a function that takes one raw in your DF as input, and returns one row. See docs here.
2. Pandas UDF, where you define a function that takes a chunk of rows in your DF as input, and return rows. See docs here.

Option 2 is most performant, but it involves operating on many rows. Still, a common approach is to define a row-level function in your PandasUDF and call it using an .apply. Take a look at the docs and try out some options.

[jose_gonzalez]

Hi @Sailaja B​,

Just a friendly follow-up. Did you saw Dan's response? Do you have any follow-up questions or can you select Dan's as best answer?

[jose_gonzalez]

Hi @Sailaja B​ ,

Just a friendly follow-up. Do you still need help or the recommendations given By Dan or Hubert help you to resolve your issue? please let us know.