Hello,
We are using a 5 worker node DLT job compute for a continuous mode streaming pipeline. The worker configuration is Standard_D4ads_v5 i.e. 4 cores so total cores across 5 workers is 20 cores.
We have wide transformation at some places in the pipeline using groupBy since we need to apply state to pandas. So at each groupBy, it is creating 200 partitions and tasks underneath. Since there are several flows in the pipeline, it is at times executing multiple stages concurrently each with 200 tasks, so the 20 core compute cluster is getting overloaded.
Hence we need to reduce the shuffle.partitions down to 20 or so however we are not finding a way to do this in DLT pipeline.
Configured following settings: spark.sql.shuffle.partitions however it is not taking effect.
Also tried spark.databricks.adaptive.autoOptimizeShuffle.enabled setting this to true however it has no effect as well and still the groupBy is using 200 partitions.
Can someone throw some suggestion as to how this shuffle partition can be changed in DLT pipeline as this is causing lot of resource issues and cpu issues with the cluster?
