cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

how to convert each row of df to array of rows(list of rows)

SailajaB
Valued Contributor III

Hi,

How to convert each row of dataframe to array of rows?

Here is our scenario , we need to pass each row of dataframe to one function as dict to apply the key level transformations. But as our data is very huge we can't use collect df.toJson().collect() to iterate over each row as it uses only driver's memory.

Please add your suggestions here.

Thank you

1 ACCEPTED SOLUTION

Accepted Solutions

Dan_Z
Honored Contributor
Honored Contributor

You have two options here:

  1. Regular PySpark UDF, where you define a function that takes one raw in your DF as input, and returns one row. See docs here.
  2. Pandas UDF, where you define a function that takes a chunk of rows in your DF as input, and return rows. See docs here.

Option 2 is most performant, but it involves operating on many rows. Still, a common approach is to define a row-level function in your PandasUDF and call it using an .apply. Take a look at the docs and try out some options.

View solution in original post

5 REPLIES 5

Hubert-Dudek
Esteemed Contributor III

To optimize performance I would write pandas vectorized function and registered it as udf . https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.pandas_udf.html

If it is one time go you can also use basic spark function like foreach https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.foreach.html

SailajaB
Valued Contributor III

@Hubert Dudek​ , Thank you for the reply.

We are new to ADB. And using the below code, looking for an optimized way to do it

dfJSONString = df.toJSON().collect()

stringList = []

  for row in dfJSONString:

    # ==== Unflatten the JSON string ==== #

    jsonString = unflatten(json.loads(row),dictreg[reg.upper()])

     stringList.append(json.dumps(jsonString))

Thank you

Dan_Z
Honored Contributor
Honored Contributor

You have two options here:

  1. Regular PySpark UDF, where you define a function that takes one raw in your DF as input, and returns one row. See docs here.
  2. Pandas UDF, where you define a function that takes a chunk of rows in your DF as input, and return rows. See docs here.

Option 2 is most performant, but it involves operating on many rows. Still, a common approach is to define a row-level function in your PandasUDF and call it using an .apply. Take a look at the docs and try out some options.

Hi @Sailaja B​,

Just a friendly follow-up. Did you saw Dan's response? Do you have any follow-up questions or can you select Dan's as best answer?

Hi @Sailaja B​ ,

Just a friendly follow-up. Do you still need help or the recommendations given By Dan or Hubert help you to resolve your issue? please let us know.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!