Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2021 03:04 AM
as @Werner Stinckens said "Spark will use multiple nodes to process the data".
If you like to use Pandas code there is Pandas API for Spark (since 3.2). What you need to do is just to import different library:
# THIS NOT: from pandas import read_csv
from pyspark.pandas import read_csv
pdf = read_csv("data.csv")Here is more info https://databricks.com/blog/2021/10/04/pandas-api-on-upcoming-apache-spark-3-2.html
My blog: https://databrickster.medium.com/