- 2044 Views
- 2 replies
- 3 kudos
How to apply Pandas functions on PySpark DataFrame?
Hi, I want to apply Pandas functions (like isna, concat, append, etc) on PySpark DataFrame in such a way that computations are done on multi-node cluster.I don't want to convert PySpark DataFrame into Pandas DataFrame since, I think, only one node is...
- 2044 Views
- 2 replies
- 3 kudos
- 3 kudos
The best is to use pandas on a spark, it is virtually interchangeable so it just different API for Spark data frameimport pyspark.pandas as ps psdf = ps.range(10) sdf = psdf.to_spark().filter("id > 5") sdf.show()
- 3 kudos
- 1213 Views
- 0 replies
- 3 kudos
Longer execution time to write into the SQL server table from Spark Dataframe
I have 8gb of XML data loaded into different dataframes, there are two dataframes which has 24 lakh and 82 lakh data to be written to a 2 SQL server tables which is taking so 2 hrs and 5 hrs of time to write it. I am using the below cluster configura...
- 1213 Views
- 0 replies
- 3 kudos