I have personally witnessed these kind of issues. 

Why these failures happen, usually as far as I have witnessed that the Driver Node might be unavailable or not responsive as you might have hit the maximum cpu or memory usage, may be your cache utilisation hit the maximum, and there could be many more reasons.

To avoid such issues I would always scheduled my workflows or Jobs with a good retry count and spread about more than 5 minutes between each retry.

Also, if the same issue is occuring every time you are running your code then you must optimize your code to read and write the data efficiently.

This worked like magic to me most of the time. EOD, it is all Availability of the compute which can never be 100 percent.