- 4651 Views
- 3 replies
- 9 kudos
Resolved! Number of partitions in Spark UI Simulator experiment
I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1​596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...
- 4651 Views
- 3 replies
- 9 kudos
Latest Reply
- 9 kudos
Hi @Bartosz Maciejewski​ Generally we arrive at the number of shuffle partitions using the following method.Input Size Data - 100 GBIdeal partition target size - 128 MBCores - 8Ideal number of partitions = (100*1028)/128 = 803.25 ~ 804To utiltize the...
- 9 kudos