topic Re: Keep long-running notebook alive when closing browser in Machine Learning

Keep long-running notebook alive when closing browser

orion216 — Thu, 12 May 2022 20:24:51 GMT

I am working with Azure Databricks jupyter notebooks and have time-consuming jobs (complex queries, model training, loops over many items, etc.).

Every time I close the browser (or step away for a long time) of some running notebook, even before the cluster time-out time, notebooks and cluster will become idle. When I open the notebook again, it appears as if it the notebook resumes, but it does not continue executing. I'd like to keep the notebook running in the background at night or when I close the browser. Any thoughts ?

Re: Keep long-running notebook alive when closing browser

Aashita — Sat, 14 May 2022 06:48:18 GMT

@Eric P , To keep the cluster active at all times,

Disable "Automatic termination"
create a notebook with simple print
"%sql select 1" commands and schedule it to run at regular intervals(avoid scheduling forever) to keep the cluster active all the time.

Also, checkout this doc- https://docs.databricks.com/clusters/clusters-manage.html?&_ga=2.26996221.1323833801.1652117370-1078332075.1651512759&_gac=1.61675614.1652215740.Cj0KCQjwmuiTBhDoARIsAPiv6L-O4RB4dnxsiFsIjzb8cmoOdzvkKkO-a4qCQ6CqgPuQhjDE0j2PfhIaAqPXEALw_wcB#automatic-termination-1

Re: Keep long-running notebook alive when closing browser

orion216 — Mon, 16 May 2022 20:01:08 GMT

Hi @Kaniz Fatma @Aashita Ramteke

There is no "Automatic termination". Are you talking about "Terminate after"

2. I look for a more sustainable solution. Your solution requires running secondary script that continuously pings the server and never stops, which translate to cost. I look for something that does not kill the job half-way and keep it alive until completion. It's a simple capability that every platform support.

Does it mean we can never run a time-consuming job or train a model without sitting there looking at the screen overnight ?

Thank you.

Kenny.

Re: Keep long-running notebook alive when closing browser

Atanu — Thu, 26 May 2022 06:40:45 GMT

I think you should try increasing the inactivity time and if the issue still persists. This should not happen actually @Eric P

Re: Keep long-running notebook alive when closing browser

Prabakar — Thu, 26 May 2022 14:55:19 GMT

@Eric P If you want the notebook to be running, why not create and schedule a job for the notebook?

https://docs.microsoft.com/en-us/azure/databricks/data-engineering/jobs/jobs#--create-a-job

Re: Keep long-running notebook alive when closing browser

Anonymous — Thu, 07 Jul 2022 16:37:55 GMT

Hey @Eric P

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best? If not, please tell us so we can help you.

Thanks!