It seems that you have only 8GB ram (probably 4-6 GB is needed for system at least) but you allocate 10GB for spark (4 GB driver + 6 GB executor).

You can allocate max in my opinion 2GB all together if your RAM is 8 GB. Maybe even 1GB as there can be also spikes in system processes.

Easier would be with docker as than you allocate your machine permanent number of ram and than spark can consume exact amount.


My blog: https://databrickster.medium.com/