Thank you for the explanation. It is insightful.

I suppose this is more like a feature request then.

Right now we cannot use repartition(10000) in a unit test because it makes that test run a lot slower. Switching to Dataset also has the same issue with slow unit test.

This makes it harder to develop on Spark because the unit test becomes too slow for a complex Spark logic.