topic Re: restarting the cluster always running doesn't free the memory? in Data Engineering

restarting the cluster always running doesn't free the memory?

jeremy98 — Tue, 04 Mar 2025 13:53:30 GMT

Hello community,

I was working on optimising the driver memory, since there are code that are not optimised for spark, and I was planning temporary to restart the cluster to free up the memory.

that could be a potential solution, since if the cluster is not working in the first few minutes of each hour it is a good moment to restart it and free up the memory. But I was looking about the standard output and seems that is not free any memory. Why this behaviour? I need to terminate and start the cluster instead of this previous operation?

Re: restarting the cluster always running doesn't free the memory?

Alberto_Umana — Tue, 04 Mar 2025 14:36:24 GMT

Hi @jeremy98,

Generally, Databricks recommends regularly restarting clusters, particularly interactive ones, for regular clean-up. Restarting or terminating and starting the cluster anew ensures stopping all processes and freeing up memory effectively, therefore the restart should actually clean it up. You can see in your cluster metrics once it is restarted.

Re: restarting the cluster always running doesn't free the memory?

jeremy98 — Tue, 04 Mar 2025 14:48:33 GMT

Thanks Alberto, for the clarification! Yes, it is true, effectively, the metric UI doubled the logs for the driver and for the worker/s. I think it is a normal behaviour.

Re: restarting the cluster always running doesn't free the memory?

Alberto_Umana — Tue, 04 Mar 2025 14:53:32 GMT

No problem, happy to assist!

Re: restarting the cluster always running doesn't free the memory?

jeremy98 — Tue, 04 Mar 2025 15:14:44 GMT

Hi, I have another question: Usually, the driver should free memory by itself, but is it possible that the driver fails to do so? Why does this happen, and what issues can arise from this behavior?

Re: restarting the cluster always running doesn't free the memory?

Alberto_Umana — Tue, 04 Mar 2025 15:22:23 GMT

Yes, it is actually possible that driver due to some reason did not free up memory.. if that happens the you will see these kind of failures:

Unresponsiveness: The driver may become unresponsive, leading to failed health checks and potential restarts or kills by watchdog mechanisms.
Frequent Restarts: Continuous memory pressure and GC overhead can cause the driver to restart frequently, leading to interruptions in job executions and degraded performance.
Out of Memory (OOM) Conditions: Eventually, the driver might run out of memory, leading to crashes and job failures with explicit OOM errors

Re: restarting the cluster always running doesn't free the memory?

jeremy98 — Tue, 04 Mar 2025 15:26:13 GMT

Exactly, thanks, Alberto! But in general, is it best practice to restart a cluster every week to prevent this issue? Or does this problem happen because the code is not well-written?

Re: restarting the cluster always running doesn't free the memory?

Alberto_Umana — Tue, 04 Mar 2025 15:32:41 GMT

It is best practice to restart the cluster regularly correct! Regularly restarting clusters can help mitigate memory leaks and accumulated GC issues.

And about if it happens because of your code, it depends on what you are doing and if you follow best practices, but would need more insights to tell.

Re: restarting the cluster always running doesn't free the memory?

jeremy98 — Wed, 05 Mar 2025 17:48:25 GMT

Hi,

The code synchronizes Databricks with PostgreSQL by identifying differences and applying INSERT, UPDATE, or DELETE operations to update PostgreSQL. The steps are as follows:

Read the source data in Databricks using a simple spark.sql query.
Read the data from PostgreSQL using the JDBC driver.
Perform a JOIN operation to identify differences.
Collect the data using .collect() (I am now trying to use .toLocalIterator()).
Chunk the data and iterate over it, executing DML operations using psycopg2 in batch (extras.execute_batch()), pushing a list of tuples with page_size=1000.
…and that’s all.

Could the issue be that psycopg2 is not an API call from Databricks, so execution is handled by the driver? Or is the .collect() operation causing a bottleneck by bringing too much data to the driver at once?

Re: restarting the cluster always running doesn't free the memory?

jeremy98 — Fri, 07 Mar 2025 10:06:57 GMT

any suggestion Mr. @Alberto_Umana ?

Re: restarting the cluster always running doesn't free the memory?

SwanandD — Fri, 02 Jan 2026 08:03:43 GMT

In my case, the disk space increase event triggered causing disk space expansion, which seems fair. But, after the cluster restart, I see new instances with increased disk space only and not the default ones. This shouldn't be the case IMO.

Re: restarting the cluster always running doesn't free the memory?

iyashk-DB — Fri, 02 Jan 2026 08:59:13 GMT

Hi @jeremy98 , collect() operation brings data to the driver and yes it can cause the memory issues that you are seeing, which can cause the cluster to be hung/ crash as well if done enough times. You may confirm these instances from the cluster event logs showing "Driver is up, but not responsive. Likely due to GC".

Looking at the screenshots, you show us that there is memory pressure. So can you try reducing the GC interval from default 30 mins to 15mins.

Please set configurations below in the Advanced options -> Spark
spark.cleaner.periodic.GC.interval 15min

These configurations will help optimizing the GC, and clear the GC objects after every 15 minutes instead of 30 minutes(default value).

Re: restarting the cluster always running doesn't free the memory?

siva-anantha — Fri, 02 Jan 2026 13:45:11 GMT

@jeremy98 : Please review the cluster's event logs to understand the trend of the GC related issues. Example in below snapshot.

Typically, productive jobs are executed using Job clusters; and they stop as soon as the work is completed. Could you please explain the purpose of using Interactive cluster?