In case somebody else runs into the same issue: After investigation from Databricks support the conclusion was that the driver's memory was overloaded ('Driver Not Responding' error message in the event log) but it can happen that we don't get the complete metric and Spark UI details when this happens. This is probably why I didn't see in the metrics tab that driver memory was exhausted. The suggested solution can be found in the following link https://kb.databricks.com/en_US/jobs/driver-unavailable