can you try without:

.set("spark.driver.memory","4g")

.set("spark.executor.memory", "6g")

It is clearly show that there is no 4gb free on driver and 6gb free on executor (you can share hardware cluster details also).

You can not also allocate 100% for spark usually as there is also other processes.

Automatic settings are recommended.