Hubert-Dudek
Databricks MVP

‎10-23-2022 02:00 PM

The best is to use pandas on a spark, it is virtually interchangeable so it just different API for Spark data frame

import pyspark.pandas as ps
 
psdf = ps.range(10)
sdf = psdf.to_spark().filter("id > 5")
sdf.show()


My blog: https://databrickster.medium.com/