- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-28-2022 05:02 AM
2GB is quite small so usually default settings are the best (so in most cases better result is not to set anything like repartition etc. and leave everything to catalyst optimizer). If you want to set custom partitioning:
- please remember about avoiding data skews (that one partition is bigger than other),
- think about avoiding data shuffles - as custom partitioning can sometimes help to avoid shuffle but often can also increase shuffle
- 1 core/thread is working on 1 partition so it have sense to have number of partitions = number of cpus but it is set automatically like that
- when there is many data shuffles you can have few hundreds partitions as it have to be shuffled and go thorough memory and network
There are many videos on databricks youtube channel about that (some really long).
In my opinion most important and easiest to analyze are data skews and it also can affect situation when everything is managed automatically (we can see it in Spark UI that 1 partition was processed in 1 second and other in 1 minute for example). It can happen when we read data which is wrong portioned (common problem is that people in sql jdbc read set lowerBound and upperBound to divide ids for example to buckets per 1000 ids in partition but with time ids are growing so it make data skew as in every partition will be 1000 rows but in last one which will take all rows above upperBond can have millions...)
My blog: https://databrickster.medium.com/