User16539034020
Databricks Employee
Databricks Employee

Hello, 

Thanks for contacting Databricks Support. 

Seems you are concern with high memory consumption in the "other" category in the driver node of a Spark cluster. As there are no logs/detail information provided, I only can address several potential causes:

  1. Memory leaks, which can gradually consume memory. These are often due to bugs or inefficient memory management in the code.
  2. As you mentioned, activities like heartbeat messages, GC, listening for job requests, hosting the Spark UI, and monitoring resources do consume memory, but it's unusual for them to take up such a large proportion.
  3. Inefficient Garbage Collection. If the GC is not configured properly or is inefficient, it might not be freeing up memory as expected.
  4. Storing extensive data in the Spark UI can also consume considerable memory.

Based on above analysis, please try with following mitigation strategies:

  1. Please refer to the below documentation for tuning Garbage collection. We can try the G1GC garbage collector with -XX:+UseG1GC.
    https://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning
    https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.htm...
  2. Utilize monitoring tools to get a more detailed view of memory usage. This can help identify specific areas where memory usage is abnormally high.
  3. Review the driver logs and stack traces for any anomalies or repeated patterns that could indicate the source of the memory usage.

View solution in original post