pyspark pandas can indeed be used instead of classic pandas. However that being said, there are some caveats. If you just copy/paste your python code, chances are real you are not using the parallel processing capabilities of spark (distributed computing). For example iterating over values of a structure. this will still execute on a single node.

So to be able to scale out your code, you should use pyspark (can be pandas) and think in vectors/sets instead of records.

Instead of iterating over records, try to apply a function in one go. that is why spark was invented.