Unable to set shuffle partitions on DLT pipeline

[PushkarDeole]

Hello,

We are using a 5 worker node DLT job compute for a continuous mode streaming pipeline. The worker configuration is Standard_D4ads_v5 i.e. 4 cores so total cores across 5 workers is 20 cores.

We have wide transformation at some places in the pipeline using groupBy since we need to apply state to pandas. So at each groupBy, it is creating 200 partitions and tasks underneath. Since there are several flows in the pipeline, it is at times executing multiple stages concurrently each with 200 tasks, so the 20 core compute cluster is getting overloaded.

Hence we need to reduce the shuffle.partitions down to 20 or so however we are not finding a way to do this in DLT pipeline.

Configured following settings: spark.sql.shuffle.partitions however it is not taking effect.

Also tried spark.databricks.adaptive.autoOptimizeShuffle.enabled setting this to true however it has no effect as well and still the groupBy is using 200 partitions.

Can someone throw some suggestion as to how this shuffle partition can be changed in DLT pipeline as this is causing lot of resource issues and cpu issues with the cluster?

[szymon_dybczak]

Hi @PushkarDeole ,

Each Delta Live Tables pipeline has two associated clusters:

• The updates cluster processes pipeline updates.

• The maintenance cluster runs daily maintenance tasks.

According to docs, if you want to configure settings at the pipeline level you can pass spark_conf option in configuration of pipeline:

https://docs.databricks.com/en/delta-live-tables/settings.html#configure-your-compute-settings

{
  "clusters": [
    {
      "label": "default",
      "autoscale": {
        "min_workers": 1,
        "max_workers": 5,
        "mode": "ENHANCED"
      }
    },
    {
      "label": "updates",
      "spark_conf": {
         "key": "value"
      }
    }
  ]
}

[PushkarDeole]

Hello @szymon_dybczak 

Let me rephrase the problem... so, I did set following parameters on the pipeline advanced configuration in order to alter the shuffle partitions however they are not taking effect and this is what the actual issue is:

spark.sql.shuffle.partitions = 20

spark.databricks.adaptive.autoOptimizeShuffle.enabled = true

During wide transformation, still see the partitions as 200