I am working in jupyter hub in a notebook. I am using pyspark dataframe for analyzing text. More precisely I am doing sentimment analysis of newspaper articles. The code works until I get to some point where the kernel is busy and after approximately 10 minutes of being busy, it switches to unknown. The operations that cause it to stop working are for example .drop() and groupBy(). The dataset has only about 25k rows. After looking at the logs I get this message:

Stage 1:> (0 + 0) / 1] 22/06/02 09:30:17 WARN TaskSetManager: Stage 1 contains a task of very large size (234399 KiB). The maximum recommended task size is 1000 KiB.

After some research I found out that it is probably due to full memory. However I am not sure how to increase it.

To build the spark application I use this code:

spark = SparkSession.builder \
        .master("local") \
        .appName("x") \
        .config("spark.driver.memory", "2g") \
        .config("spark.executor.memory", "12g") \
        .getOrCreate()
sc = spark.sparkContext
sqlContext = SQLContext(sc)

Any ideas for the kernel to stop changing to "Unknown" or somehow free the memory? Note: I am not using RDDs just spark dataframes

I am sharing my notebook. This project is for my thesis and I am desperate to get the code working. Would be extremely thankful for any help!