cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Jobs running forever in Spark UI

User16869510359
Esteemed Contributor

On the Spark UI, Jobs are running forever. But my notebook already completed the operations. Why the resources are wasted

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

This happens if the Spark driver is missing events. The jobs/task are not running. The Spark UI is reporting incorrect stats. This can be treated as a harmless UI issue. If you continue to see the issue consistently, then it might be good to review what is causing the Spark driver to be memory intensive and lose the Spark events. 

View solution in original post

1 REPLY 1

User16869510359
Esteemed Contributor

This happens if the Spark driver is missing events. The jobs/task are not running. The Spark UI is reporting incorrect stats. This can be treated as a harmless UI issue. If you continue to see the issue consistently, then it might be good to review what is causing the Spark driver to be memory intensive and lose the Spark events. 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.