Given the provided context, the suggestion is to capture thread dumps from both the Spark Driver and any Active Executor when the task seems to be hung. Ideally, you should also be able to find in the Spark logs for the active executor with the hung task, coming from a HangTaskDetector class which also captures the threaddumps that are ready for analysis. These will give you some insight on why is the task hung (...or slowly progressing). So you want to focus on the executor logs instead of Driver's.

The output log shared is not very relevant, it only shows two instances of a spark driver initialization followed by the cancellation command.