topic how to convert each row of df to array of rows(list of rows) in Data Engineering

how to convert each row of df to array of rows(list of rows)

SailajaB — Sun, 27 Feb 2022 14:15:09 GMT

Hi,

How to convert each row of dataframe to array of rows?

Here is our scenario , we need to pass each row of dataframe to one function as dict to apply the key level transformations. But as our data is very huge we can't use collect df.toJson().collect() to iterate over each row as it uses only driver's memory.

Please add your suggestions here.

Thank you

Re: how to convert each row of df to array of rows(list of rows)

Hubert-Dudek — Sun, 27 Feb 2022 14:38:19 GMT

To optimize performance I would write pandas vectorized function and registered it as udf . https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.pandas_udf.html

If it is one time go you can also use basic spark function like foreach https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.foreach.html

Re: how to convert each row of df to array of rows(list of rows)

SailajaB — Mon, 28 Feb 2022 10:19:28 GMT

@Hubert Dudek , Thank you for the reply.

We are new to ADB. And using the below code, looking for an optimized way to do it

dfJSONString = df.toJSON().collect()

stringList = []

for row in dfJSONString:

# ==== Unflatten the JSON string ==== #

jsonString = unflatten(json.loads(row),dictreg[reg.upper()])

stringList.append(json.dumps(jsonString))

Thank you

Re: how to convert each row of df to array of rows(list of rows)

Dan_Z — Wed, 04 May 2022 16:10:27 GMT

You have two options here:

Regular PySpark UDF, where you define a function that takes one raw in your DF as input, and returns one row. See docs here.
Pandas UDF, where you define a function that takes a chunk of rows in your DF as input, and return rows. See docs here.

Option 2 is most performant, but it involves operating on many rows. Still, a common approach is to define a row-level function in your PandasUDF and call it using an .apply. Take a look at the docs and try out some options.

Re: how to convert each row of df to array of rows(list of rows)

jose_gonzalez — Tue, 10 May 2022 16:39:35 GMT

Hi @Sailaja B ,

Just a friendly follow-up. Do you still need help or the recommendations given By Dan or Hubert help you to resolve your issue? please let us know.

Re: how to convert each row of df to array of rows(list of rows)

jose_gonzalez — Tue, 07 Jun 2022 16:20:21 GMT

Hi @Sailaja B,

Just a friendly follow-up. Did you saw Dan's response? Do you have any follow-up questions or can you select Dan's as best answer?