-werners-
Esteemed Contributor III

Spark is a distributed data processing framework. For it to shine, you need multiple machines (VMs or physical). Otherwise it is no better than pandas etc (in local mode on a single node).

So to start using spark, you should either connect to an existing spark cluster (if there is a cluster available for you) or, and that might be the easiest way: sign up for Databricks Community Edition and start using Databricks.

Mind that Community Edition is limited in functionality, but still very useful.

https://docs.databricks.com/getting-started/quick-start.html

If you cannot do either, stop using pyspark and focus on pure python code.

You can still run into memory issues though as you run code locally.