How to auto‑terminate DLT-managed clusters after pipeline execution?

[anusha98]

We have Data bricks Jobs that run a combination of DLT pipelines and notebook tasks.
For the notebook tasks, we use a job cluster, and we are able to auto‑terminate it after execution using auto termination: 10

However, the DLT-managed clusters behave differently — after the pipeline completes, the DLT cluster continues running for up to 60 minutes before shutting down. This results in unnecessary additional cost.

Is there any way to reduce the idle time or auto‑terminate DLT-managed clusters sooner, similar to job clusters? Or any configuration available to control the DLT cluster shutdown time?

Any guidance or recommended best practices would be appreciated.

[Louis_Frolio · Accepted solution]

@anusha98 , Make sure you are running the pipeline in Production mode and not Development mode.

https://docs.databricks.com/aws/en/ldp/updates#optimize-execution

[SteveOstrowski]

Hi @anusha98,

The behavior you are seeing, where the cluster stays running for a long time after pipeline completion, is almost certainly because your pipeline is running in Development mode. In Development mode, the default cluster shutdown delay is 2 hours. In Production mode, the default is 0 seconds, meaning the cluster terminates immediately after the pipeline update finishes.

Here is how to address this:

OPTION 1: SWITCH TO PRODUCTION MODE

If your pipeline is ready for production workloads (scheduled runs, no active debugging), toggle Development mode off in the pipeline settings UI, or set it in your pipeline JSON configuration:

"development": false

In Production mode, the cluster shuts down with 0 seconds of delay by default, which is what you want for cost optimization.

OPTION 2: CONFIGURE THE SHUTDOWN DELAY EXPLICITLY

If you need to stay in Development mode (for faster iteration, cluster reuse between updates, etc.) but want to reduce the idle time, use the pipelines.clusterShutdown.delay configuration parameter. You can set this in the pipeline configuration section:

{
  "configuration": {
    "pipelines.clusterShutdown.delay": "60s"
  }
}

This tells the pipeline to shut down the cluster 60 seconds after the update completes, instead of the default 2 hours. You can set it to any value that makes sense for your workflow, even "0s" if you want immediate shutdown.

WHY THIS IS DIFFERENT FROM JOB CLUSTER AUTO-TERMINATION

Lakeflow Spark Declarative Pipelines (SDP), previously known as DLT, manage their own compute lifecycle. You cannot use the standard autotermination_minutes setting from cluster policies or job cluster configs. Attempting to set autotermination_minutes in a compute policy for an SDP pipeline will result in an error. The pipelines.clusterShutdown.delay setting is the correct and only mechanism for controlling this behavior.

QUICK SUMMARY

Development mode default shutdown delay: 2 hours
Production mode default shutdown delay: 0 seconds
Custom override: pipelines.clusterShutdown.delay (e.g., "60s", "0s", "5m")

DOCUMENTATION REFERENCES

Configure compute for a pipeline:
https://docs.databricks.com/en/delta-live-tables/configure-compute.html

Pipeline settings:
https://docs.databricks.com/en/delta-live-tables/settings.html

Run a pipeline update:
https://docs.databricks.com/en/delta-live-tables/updates.html

Note: "DLT" (Delta Live Tables) has been renamed to Lakeflow Spark Declarative Pipelines (SDP). The configuration parameters and behavior remain the same.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.