- 6062 Views
- 1 replies
- 0 kudos
Resolved! Ideal number and size of partitions
Spark by default uses 200 partitions when doing transformations. The 200 partitions might be too large if a user is working with small data, hence it can slow down the query. Conversely, the 200 partitions might be too small if the data is big. So ho...
- 6062 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
You could tweak the default value 200 by changing spark.sql.shuffle.partitions configuration to match your data volume. Here is a sample python code for calculating the valueHowever if you have multiple workloads with different data volumes, instead ...
- 0 kudos