The error is most likely caused by the row-per-row processing (due to the use of spacy).
Like that you bypass the parallel processing capabilities of spark.
A solution would by to not use a loop.  But in your case, using spacy, that does not seem possible ( I looked online but everybody seems to use a UDF and an iterator so that won't solve your issue).
Is it an option to use another NLP library that can run on pyspark like SparkNLP?