Miguel_Suarez
Databricks Employee
Databricks Employee

Hi @Nate-Haines,

Regularly restarting long-running shared clusters will help release memory on the cluster. The Driver maintains the state information for the all notebooks attached to the cluster and as well as maintaining the SparkContext: leaving these notebooks idle on the cluster seems can cause the Driver not being able to efficiently release memory.

Hope this helps!

Best,

Miguel