cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Bartek
by • Contributor
  • 1046 Views
  • 2 replies
  • 1 kudos

Resolved! Spark UI simulator is not available online

About 2 weeks ago I started course on "Optimizing Apache Spark on Databricks" from official Databricks academy. It is heavily based on Spark UI simulator experiments that were available here: https://www.databricks.training/spark-ui-simulator and for...

Data Engineering
Reply
  • 1046 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
LandanG
Honored Contributor
  • 1 kudos

Hi @Bartosz Maciejewski​ ,Can you try loading the website without https and instead just http like http://www.databricks.training/spark-ui-simulator/ ?

  • 1 kudos
1 More Replies
Bartek
by • Contributor
  • 2538 Views
  • 3 replies
  • 7 kudos

Resolved! Number of partitions in Spark UI Simulator experiment

I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1​596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...

obraz
Data Engineering
Reply
  • 2538 Views
  • 3 replies
  • 7 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 7 kudos

Hi @Bartosz Maciejewski​ Generally we arrive at the number of shuffle partitions using the following method.Input Size Data - 100 GBIdeal partition target size - 128 MBCores - 8Ideal number of partitions = (100*1028)/128 = 803.25 ~ 804To utiltize the...

  • 7 kudos
2 More Replies
Labels