Exactly. You can easy verify that as commits are written to separate files in delta log.

Regarding:

.config('spark.databricks.adaptive.autoOptimizeShuffle.enabled', 'true')

and other spark optimization solutions please watch databricks video about that https://www.youtube.com/watch?v=daXEp4HmS-E