Spark by default uses 200 partitions when doing transformations. The 200 partitions might be too large if a user is working with small data, hence it can slow down the query. Conversely, the 200 partitions might be too small if the data is big. So how do I figure out what the ideal partition size should be?

Ideal partition size is expected to be 128 MB to 1 GB. How do I go about making sure that partition size falls in this category?