Hi Andrew I would suggest starting with the Spark UI to identify which stages are taking the most time. Large data shuffles and uneven partitions can often slow down Spark jobs. Try filtering data early, selecting only the columns you need, and check...
