- 1864 Views
- 2 replies
- 1 kudos
Resolved! Spark UI simulator is not available online
About 2 weeks ago I started course on "Optimizing Apache Spark on Databricks" from official Databricks academy. It is heavily based on Spark UI simulator experiments that were available here: https://www.databricks.training/spark-ui-simulator and for...
- 1864 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Bartosz Maciejewski​ ,Can you try loading the website without https and instead just http like http://www.databricks.training/spark-ui-simulator/ ?
- 1 kudos
- 4831 Views
- 3 replies
- 9 kudos
Resolved! Number of partitions in Spark UI Simulator experiment
I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1​596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...
- 4831 Views
- 3 replies
- 9 kudos
- 9 kudos
Hi @Bartosz Maciejewski​ Generally we arrive at the number of shuffle partitions using the following method.Input Size Data - 100 GBIdeal partition target size - 128 MBCores - 8Ideal number of partitions = (100*1028)/128 = 803.25 ~ 804To utiltize the...
- 9 kudos