However, this fix is not perfect. The same error occurs but significantly less often. I have also switched the cluster settings as such:

r5d.8xlarge - worker/driver
worker count 1-16 1 on-demand 15 spot

spark config:
spark.driver.maxResultSize 16g