clearCache@Maarten van Raaij​ :

Reason for calling unpersist() after clearCache() ->

When you call spark.catalog.clearCache(), it clears the cache of all cached tables and DataFrames in Spark. However, it's important to note that the clearCache()

method only removes the metadata associated with the cached tables and DataFrames, and not the actual cached data itself. The actual cached data remains in memory until it is either evicted due to memory pressure or until it is explicitly unpersisted using the unpersist() method.