02-03-2022 01:38 PM
Currently, I am running a cluster that is set to terminate after 60 minutes of inactivity. However, in one of my notebooks, one of the cells is still running. How can I prevent this from happening, if want my notebook to run overnight without monitoring it and why is this happening?
02-03-2022 02:07 PM
@Kevin Kim As mentioned in the docs at https://docs.databricks.com/clusters/clusters-manage.html#automatic-termination-1 an autoterminating cluster may be terminated while it is running commands. From docs > The auto termination feature monitors only Spark jobs, not user-defined local processes. Therefore, if all Spark jobs have completed, a cluster may be terminated even if local processes are running.
If you want to keep the cluster active all the time, either you can disable "Automatic termination"(if allowed) or create a notebook with simple print or "%sql select 1" commands and schedule it to run at regular intervals(avoid scheduling forever) to keep the cluster active all the time.
Also, did you explore scheduling the notebook as a job? since auto termination is applicable only for all-purpose clusters.
***Note: Idle clusters continue to accumulate DBU and cloud instance charges during the inactivity period before termination.***
02-03-2022 02:07 PM
@Kevin Kim As mentioned in the docs at https://docs.databricks.com/clusters/clusters-manage.html#automatic-termination-1 an autoterminating cluster may be terminated while it is running commands. From docs > The auto termination feature monitors only Spark jobs, not user-defined local processes. Therefore, if all Spark jobs have completed, a cluster may be terminated even if local processes are running.
If you want to keep the cluster active all the time, either you can disable "Automatic termination"(if allowed) or create a notebook with simple print or "%sql select 1" commands and schedule it to run at regular intervals(avoid scheduling forever) to keep the cluster active all the time.
Also, did you explore scheduling the notebook as a job? since auto termination is applicable only for all-purpose clusters.
***Note: Idle clusters continue to accumulate DBU and cloud instance charges during the inactivity period before termination.***
02-03-2022 04:18 PM
Can you just turn off the auto turn off?
02-03-2022 11:41 PM
indeed, uncheck the 'Terminate after x minutes of inactivity' flag
02-05-2022 05:36 AM
Hi @Kevin Kim , Were you able to resolve your question through the above-provided solutions?
02-10-2022 06:13 AM
If a cell is already running ( I assume it's a streaming operation), then I think it doesn't mean that the cluster is inactive. The cluster should be running if a cell is running on it.
On the other hand, if you want to keep running your clusters for a specific period of time (say 10pm to 8am), then you can schedule a cron job to invoke at 55th minute to run a basic command in a simple notebook on that very cluster. You can schedule the job execution using a cron expression.
Also, using cluster API, you can monitor if the cluster is running or not.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.