Re: how to convert each row of df to array of rows... - Databricks Community

To optimize performance I would write pandas vectorized function and registered it as udf . https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.pandas_udf.html

If it is one time go you can also use basic spark function like foreach https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.foreach.html

My blog: https://databrickster.medium.com/