- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2022 03:38 AM
Spark is a distributed data processing framework. For it to shine, you need multiple machines (VMs or physical). Otherwise it is no better than pandas etc (in local mode on a single node).
So to start using spark, you should either connect to an existing spark cluster (if there is a cluster available for you) or, and that might be the easiest way: sign up for Databricks Community Edition and start using Databricks.
Mind that Community Edition is limited in functionality, but still very useful.
https://docs.databricks.com/getting-started/quick-start.html
If you cannot do either, stop using pyspark and focus on pure python code.
You can still run into memory issues though as you run code locally.