-werners-
Esteemed Contributor III

every storage system has its limits concerning throughput. It might be the disks or the nics.

Also when you talk about scaling: the power of spark lies not in the capability to run a ton of jobs in parallel, but by dividing a workload into tasks and process those tasks in parallel on multiple workers.