"DataFrame increases the partition number to 200 automatically when Spark operation performs data shuffling (join(), union(), aggregation functions). ."

it is from https://sparkbyexamples.com/spark/spark-shuffle-partitions/

4 workers with 2 cores in many cases will give 8 partitions (but not always 🙂 ). You can always validate it by running YourDataframe.rdd.getNumPartitions

There is more on mentioned link.

That video is also good to watch (although it is more advanced) https://www.youtube.com/watch?v=daXEp4HmS-E