Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 11:55 AM
On the Spark UI, Jobs are running forever. But my notebook already completed the operations. Why the resources are wasted
Labels:
- Labels:
-
Spark ui
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 11:55 AM
This happens if the Spark driver is missing events. The jobs/task are not running. The Spark UI is reporting incorrect stats. This can be treated as a harmless UI issue. If you continue to see the issue consistently, then it might be good to review what is causing the Spark driver to be memory intensive and lose the Spark events.