Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-19-2024 12:54 AM
Hi @genevive_mdonça ,
You can use following formula to calculate optimal count of partitions based on size of input data and target partition size:
Input Stage Data 300GB
Target Size = 200MB
Optimal Count of Partitions = 300,000 MB / 200 = 1500 partitions
Spark.conf.set(“spark.sql.shuffle.partitions”,1500)
Remember, usually partitions should not be less than number of cores
Though, by default Adaptive Query Execution (AQE) should be enabled and Spark can dynamically optimize the partition size based on runtime statistics