Databricks Community

SailajaB · ‎02-27-2022

Hi,

How to convert each row of dataframe to array of rows?

Here is our scenario , we need to pass each row of dataframe to one function as dict to apply the key level transformations. But as our data is very huge we can't use collect df.toJson().collect() to iterate over each row as it uses only driver's memory.

Please add your suggestions here.

Thank you

Dan_Z · ‎05-04-2022

You have two options here:

Regular PySpark UDF, where you define a function that takes one raw in your DF as input, and returns one row. See docs here.
Pandas UDF, where you define a function that takes a chunk of rows in your DF as input, and return rows. See docs here.

Option 2 is most performant, but it involves operating on many rows. Still, a common approach is to define a row-level function in your PandasUDF and call it using an .apply. Take a look at the docs and try out some options.

View solution in original post

Hubert-Dudek · ‎02-27-2022

To optimize performance I would write pandas vectorized function and registered it as udf . https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.pandas_udf.html

If it is one time go you can also use basic spark function like foreach https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.foreach.html

SailajaB · ‎02-28-2022

@Hubert Dudek , Thank you for the reply.

We are new to ADB. And using the below code, looking for an optimized way to do it

dfJSONString = df.toJSON().collect()

stringList = []

for row in dfJSONString:

# ==== Unflatten the JSON string ==== #

jsonString = unflatten(json.loads(row),dictreg[reg.upper()])

stringList.append(json.dumps(jsonString))

Thank you

Dan_Z · ‎05-04-2022

You have two options here:

Regular PySpark UDF, where you define a function that takes one raw in your DF as input, and returns one row. See docs here.
Pandas UDF, where you define a function that takes a chunk of rows in your DF as input, and return rows. See docs here.

Option 2 is most performant, but it involves operating on many rows. Still, a common approach is to define a row-level function in your PandasUDF and call it using an .apply. Take a look at the docs and try out some options.