- 4343 Views
- 3 replies
- 1 kudos
Resolved! Spark UI simulator is not available online
About 2 weeks ago I started course on "Optimizing Apache Spark on Databricks" from official Databricks academy. It is heavily based on Spark UI simulator experiments that were available here: https://www.databricks.training/spark-ui-simulator and for...
- 4343 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @LandanG The link is still not working. Any idea when it will be live again?http://www.databricks.training/spark-ui-simulator/GitHub - databricks-academy/spark-ui-simulator: Apache Spark UI simulator with 30+ educational experiments, pre-recorded ...
- 1 kudos
- 12555 Views
- 3 replies
- 10 kudos
Resolved! Number of partitions in Spark UI Simulator experiment
I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1​596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...
- 12555 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi @Bartosz Maciejewski​ Generally we arrive at the number of shuffle partitions using the following method.Input Size Data - 100 GBIdeal partition target size - 128 MBCores - 8Ideal number of partitions = (100*1028)/128 = 803.25 ~ 804To utiltize the...
- 10 kudos