Jobs running forever in Spark UI

brickster_2018
Databricks Employee
Databricks Employee

On the Spark UI, Jobs are running forever. But my notebook already completed the operations. Why the resources are wasted

brickster_2018
Databricks Employee
Databricks Employee

This happens if the Spark driver is missing events. The jobs/task are not running. The Spark UI is reporting incorrect stats. This can be treated as a harmless UI issue. If you continue to see the issue consistently, then it might be good to review what is causing the Spark driver to be memory intensive and lose the Spark events. 

View solution in original post