cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Keep long-running notebook alive when closing browser

orion216
New Contributor II

I am working with Azure Databricks jupyter notebooks and have time-consuming jobs (complex queries, model training, loops over many items, etc.).

Every time I close the browser (or step away for a long time) of some running notebook, even before the cluster time-out time, notebooks and cluster will become idle. When I open the notebook again, it appears as if it the notebook resumes, but it does not continue executing. I'd like to keep the notebook running in the background at night or when I close the browser. Any thoughts ?

1 ACCEPTED SOLUTION

Accepted Solutions

Atanu
Databricks Employee
Databricks Employee

I think you should try increasing the inactivity time and if the issue still persists. This should not happen actually @Eric P​ 

View solution in original post

5 REPLIES 5

Aashita
Databricks Employee
Databricks Employee

@Eric P​ , To keep the cluster active at all times,

  • Disable "Automatic termination"
  • create a notebook with simple print
  • "%sql select 1" commands and schedule it to run at regular intervals(avoid scheduling forever) to keep the cluster active all the time.
  •  

 Also, checkout this doc- https://docs.databricks.com/clusters/clusters-manage.html?&_ga=2.26996221.1323833801.1652117370-1078...

orion216
New Contributor II

Hi @Kaniz Fatma​ @Aashita Ramteke​ 

  1. There is no "Automatic termination". Are you talking about "Terminate after"

image.png2. I look for a more sustainable solution. Your solution requires running secondary script that continuously pings the server and never stops, which translate to cost. I look for something that does not kill the job half-way and keep it alive until completion. It's a simple capability that every platform support.

Does it mean we can never run a time-consuming job or train a model without sitting there looking at the screen overnight ?

Thank you.

Kenny.

Atanu
Databricks Employee
Databricks Employee

I think you should try increasing the inactivity time and if the issue still persists. This should not happen actually @Eric P​ 

Prabakar
Databricks Employee
Databricks Employee

@Eric P​ If you want the notebook to be running, why not create and schedule a job for the notebook?

https://docs.microsoft.com/en-us/azure/databricks/data-engineering/jobs/jobs#--create-a-job

Anonymous
Not applicable

Hey @Eric P​ 

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best? If not, please tell us so we can help you.

Thanks!

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now