I am actually trying to extract the adjective and noun phrases from the text column in spark data frame for which I've written the udf and applying on cleaned text column. However, I am getting this error.

from pyspark.sql.functions import udf

from pyspark.sql.types import ArrayType, StringType

import spacy

# Load spacy model

nlp = spacy.load("en_core_web_sm")

# Define UDF to extract key phrases

def extract_adjective_noun_key_phrases(text):

  doc = nlp(text)

  key_phrases = []

  for token in doc:

    if (token.pos_ == "ADJ" and token.nbor().pos_ == "NOUN") or (token.pos_ == "NOUN" and token.nbor().pos_ == "ADJ"):

      key_phrases.append(token.text + " " + token.nbor().text)

  return key_phrases

extract_adjective_noun_key_phrases_udf = udf(extract_adjective_noun_key_phrases, ArrayType(StringType()))

# Apply UDF to text column in DataFrame

pqms = pqms.withColumn("adjective_noun_key_phrases", extract_adjective_noun_key_phrases_udf(col("cleaned_text")))

# Print resulting DataFrame

display(pqms)

The expected output here to extract the phrases and create a new column for the same in spark data frame. Any help or suggestion on this will be a great help.

Thanks,