Tharun-Kumar
Databricks Employee
Databricks Employee

@wojciech_jakubo 

About the first question, driver memory utilization is high and we could see multiple cycles of high utlization. The primary reason behind this is, even if a cluster is idle, driver has to perform multiple operations to keep the cluster active and ready for processing. Some of the activities are

  • heart beat messages
  • gc
  • listening for job requests
  • hosting spark ui
  • monitoring resources

This happens in intervals and this is the reason behind memory utilization happening in cycles on the driver.