szymon_dybczak
Esteemed Contributor III

Hi @PushkarDeole ,

Each Delta Live Tables pipeline has two associated clusters:

  • The updates cluster processes pipeline updates.

  • The maintenance cluster runs daily maintenance tasks.

According to docs, if you want to configure settings at the pipeline level you can pass spark_conf option in configuration of pipeline:


https://docs.databricks.com/en/delta-live-tables/settings.html#configure-your-compute-settings

 

 

{
  "clusters": [
    {
      "label": "default",
      "autoscale": {
        "min_workers": 1,
        "max_workers": 5,
        "mode": "ENHANCED"
      }
    },
    {
      "label": "updates",
      "spark_conf": {
         "key": "value"
      }
    }
  ]
}