topic Re: What is the proper way to import the new pyspark.pandas library? in Data Engineering

What is the proper way to import the new pyspark.pandas library?

cconnell — Wed, 27 Oct 2021 17:00:17 GMT

I am moving an existing, working pandas program into Databricks. I want to use the new pyspark.pandas library, and change my code as little as possible. It appears that I should do the following:

1) Add from pyspark import pandas as ps at the top

2) Change all occurrences of pd.pandas_function to ps.pandas_function

Is this correct?

Re: What is the proper way to import the new pyspark.pandas library?

Dan_Z — Thu, 28 Oct 2021 03:21:55 GMT

Yes- that is a great start. Right now code coverage is at 83%, and we are shooting for 90%. Please file an issue if you find a gap you need filled. Please note- pandas on Spark (koalas) does some interesting things with the way it distributes the indexes of large dataframes. It may good to review starting at 20:20 here: https://databricks.com/session_na20/koalas-making-an-easy-transition-from-pandas-to-apache-spark

Re: What is the proper way to import the new pyspark.pandas library?

Hubert-Dudek — Thu, 28 Oct 2021 09:28:31 GMT

import pyspark.pandas as ps but as code coverage is 83% as @Dan Zafar said it can not guarantee that your old code will work. You can find more here on that blogpost https://databricks.com/blog/2021/10/04/pandas-api-on-upcoming-apache-spark-3-2.html

Re: What is the proper way to import the new pyspark.pandas library?

cconnell — Thu, 28 Oct 2021 13:38:25 GMT

Thank you. I am porting this code and will file issues as I see them… https://medium.com/@chuck.connell.3/vaccines-vs-mortality-correlation-at-the-county-level-922a10236a93

Re: What is the proper way to import the new pyspark.pandas library?

cconnell — Thu, 28 Oct 2021 13:40:58 GMT

I have read that blog. It was helpful but implied that I can move code from laptop pandas to spark pandas by changing one line of code, which does not seem true.

Re: What is the proper way to import the new pyspark.pandas library?

Dan_Z — Thu, 28 Oct 2021 15:24:29 GMT

That''s right, expect some minor refactoring.

Re: What is the proper way to import the new pyspark.pandas library?

Anonymous — Sat, 30 Oct 2021 10:04:31 GMT

Make sure to use the 10.0 Runtime which includes Spark 3.2

Re: What is the proper way to import the new pyspark.pandas library?

cconnell — Sat, 30 Oct 2021 12:40:28 GMT

Yes, did that. I am now porting the code, finding workarounds to problems, and keeping a list of issues. Will write it all up as an article on Medium.

Re: What is the proper way to import the new pyspark.pandas library?

Anonymous — Sat, 30 Oct 2021 21:00:52 GMT

Cool, make sure to link it here when you're finished.

Re: What is the proper way to import the new pyspark.pandas library?

cconnell — Sat, 30 Oct 2021 22:30:25 GMT

Yes. Do you want me to create official issues at apache/spark, or let someone on your team do it from my notes?

Re: What is the proper way to import the new pyspark.pandas library?

Anonymous — Sat, 30 Oct 2021 23:32:45 GMT

Official issues is the best way to go. That way you can reference them in the future and show your work off!

Re: What is the proper way to import the new pyspark.pandas library?

cconnell — Mon, 01 Nov 2021 15:01:44 GMT

I created some issues, such as this one. Please let me know if I did anything wrong, so I can fix them. Thanks!

https://issues.apache.org/jira/browse/SPARK-37180