Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-14-2023 10:36 PM
About the first question, driver memory utilization is high and we could see multiple cycles of high utlization. The primary reason behind this is, even if a cluster is idle, driver has to perform multiple operations to keep the cluster active and ready for processing. Some of the activities are
- heart beat messages
- gc
- listening for job requests
- hosting spark ui
- monitoring resources
This happens in intervals and this is the reason behind memory utilization happening in cycles on the driver.