Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-30-2024 01:25 AM
Hi @PushkarDeole ,
Each Delta Live Tables pipeline has two associated clusters:
The updates cluster processes pipeline updates.
The maintenance cluster runs daily maintenance tasks.
According to docs, if you want to configure settings at the pipeline level you can pass spark_conf option in configuration of pipeline:
https://docs.databricks.com/en/delta-live-tables/settings.html#configure-your-compute-settings
{
"clusters": [
{
"label": "default",
"autoscale": {
"min_workers": 1,
"max_workers": 5,
"mode": "ENHANCED"
}
},
{
"label": "updates",
"spark_conf": {
"key": "value"
}
}
]
}