If I write pandas code using koalas and have photon enabled, will my pandas code run on photon?

User16752240150 — Fri, 04 Jun 2021 19:35:28 GMT

Re: If I write pandas code using koalas and have photon enabled, will my pandas code run on photon?

holly — Mon, 08 Apr 2024 10:30:16 GMT

Hi there! Appreciate this reply is 3 years later than it was originally asked, but people might be coming across it still. A few things:

Koalas was deprecated in spark 3.2 (runtime 10.4). Instead, the recommendation is to use pandas on spark with `import pyspark.pandas as ps` You can find a link here to the spark migration guide, and here for more usage
As of writing, photon works with SQL and equivalent DataFrame API statements. So SQL-ish statements like filter, join, and aggregates will work, but more complex ones for analytics or data science it won't.
In the future, there may be more functionality bought out, but keep in mind that UDFs and RDDs are unlikely to ever work with photon as they bypass sparks catalyst optimizer which is needed for it to work.