anardinelli
Databricks Employee
Databricks Employee

Hey @jimcast how are you?

You can check the internals and have a good hint of what's happening using the SparkUI. Filter and select the jobs that are taking the longest and check what is being requested on the SQL/Data Frame tab, as well as their plans. 

If your data is public, please also share more details (such as logs, prints and dumps) so we can better help you with.

Best,

Alessandro