@Maarten van Raaij​ :

• About the error on ``getCache()`` -> The error message you are receiving suggests that the Spark context does not have a .getCache() method available. This may be because the method is deprecated . Instead, can you try and use the SparkSession.catalog.clearCache() method to clear the cached data.

Example:

from pyspark.sql import SparkSession
 
# create a Spark session
spark = SparkSession.builder.appName("ExampleApp").getOrCreate()
 
# cache a DataFrame
df = spark.read.csv("data.csv")
df.cache()
 
# clear the cache
spark.catalog.clearCache()
 
# unpersist the DataFrame from memory
df.unpersist()

Note that the cache() method on the DataFrame is used to cache the data in memory. The unpersist() method is used to remove the data from memory after it is no longer needed.

• Why cache must be showing up in sparkUI

It's possible that you are using the wrong Spark context to access the cached RDD. If you cache an RDD using the SparkContext object, you need to use the same object to retrieve the cached RDD later. Similarly, if you cache a DataFrame using the SparkSession object, you need to use the same object to retrieve the cached DataFrame later. If you are using the

sql_context object to access the cached RDD, it may not be able to find the cached RDD because it was cached using a different Spark context.