Any known issue with interactive Shared Cluster Driver Memory Cleanup

filipniziol — Tue, 11 Mar 2025 13:56:40 GMT

I am experiencing memory leaks on a Standard (formerly shared) interactive cluster:

1. We run jobs regularly on the cluster
2. After each job completes, driver memory usage continues to increase, suggesting resources aren't fully released
3. Eventually, the cluster crashes with "Out of Memory" errors after ~1/2 day of executions

Details:
- Using DBR 15.4
- The same jobs ran for weeks without issues on a "No isolation shared" cluster
- We are using ThreadPoolExecutor to parallelize notebook/function runs
- The memory usage grows consistently after each job execution
- We can see occasional drops in memory usage (indicating garbage collection is happening), but the overall trend is still increasing memory consumption over time

Questions:
1. Is this a known issue with DBR 15.4 on Standard (formerly shared) interactive clusters?
2. Are there specific settings or configurations that might help prevent this memory leak?
3. Are there recommended methods to monitor and diagnose driver memory usage when direct JVM access is restricted?

Any guidance would be appreciated!

I noticed similar posts
https://community.databricks.com/t5/data-engineering/restarting-the-cluster-always-running-doesn-t-free-the-memory/td-p/111710
https://community.databricks.com/t5/data-engineering/drive-memory-utilization-cleanup/td-p/106519:

Re: Any known issue with interactive Shared Cluster Driver Memory Cleanup

jdrum — Fri, 21 Mar 2025 11:38:54 GMT

Similar problem. Guess when we switched to 15.4 from 14.3 😞

Re: Any known issue with interactive Shared Cluster Driver Memory Cleanup

jdrum — Fri, 21 Mar 2025 11:52:15 GMT

but maybe fixed today?

Re: Any known issue with interactive Shared Cluster Driver Memory Cleanup

Alberto_Umana — Fri, 21 Mar 2025 12:22:10 GMT

Hello Team,

I'll check internally if any known issue reported.

topic Re: Any known issue with interactive Shared Cluster Driver Memory Cleanup in Data Engineering

Any known issue with interactive Shared Cluster Driver Memory Cleanup

Re: Any known issue with interactive Shared Cluster Driver Memory Cleanup

Re: Any known issue with interactive Shared Cluster Driver Memory Cleanup

Re: Any known issue with interactive Shared Cluster Driver Memory Cleanup