- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-03-2022 02:07 PM
@Kevin Kim As mentioned in the docs at https://docs.databricks.com/clusters/clusters-manage.html#automatic-termination-1 an autoterminating cluster may be terminated while it is running commands. From docs > The auto termination feature monitors only Spark jobs, not user-defined local processes. Therefore, if all Spark jobs have completed, a cluster may be terminated even if local processes are running.
If you want to keep the cluster active all the time, either you can disable "Automatic termination"(if allowed) or create a notebook with simple print or "%sql select 1" commands and schedule it to run at regular intervals(avoid scheduling forever) to keep the cluster active all the time.
Also, did you explore scheduling the notebook as a job? since auto termination is applicable only for all-purpose clusters.
***Note: Idle clusters continue to accumulate DBU and cloud instance charges during the inactivity period before termination.***