Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-06-2021 12:51 PM
I'm new to Spark and trying to understand how some of its components work.
I understand that once the data is loaded into the memory of separate nodes, they process partitions in parallel, within their own memory (RAM).
But I'm wondering whether the initial partition loads into memory are done in parallel as well? AFAIK some SSDs allow for concurrent reads, but not sure whether that applies here.
Also, what exactly is partitioning in the context of Spark? Does the original file get split into different smaller files, or each nodes reads from a certain begin_byte to end_byte?
Labels:
- Labels:
-
Parallelism
-
Partitioning
-
Read data