Hubert-Dudek
Databricks MVP

as @Werner Stinckens​ said "Spark will use multiple nodes to process the data".

If you like to use Pandas code there is Pandas API for Spark (since 3.2). What you need to do is just to import different library:

# THIS NOT: from pandas import read_csv
from pyspark.pandas import read_csv
pdf = read_csv("data.csv")

Here is more info https://databricks.com/blog/2021/10/04/pandas-api-on-upcoming-apache-spark-3-2.html


My blog: https://databrickster.medium.com/