unpersist doesn't clear
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-07-2023 07:37 AM
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
from pyspark.storagelevel import StorageLevel
spark = SparkSession.builder.appName('TEST').config('spark.ui.port','4098').enableHiveSupport().getOrCreate()
df4 = spark.sql(' \
select * from hive_schema.table_name limit 1')
print("query completed " )
df4.unpersist()
df4.count()
df4.show()
I have execute above code to clear the dataframe release the memory. However, df4.show() still works and shows the data. Could you please help me with right method to free memory occupied by a spark DF please ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-01-2023 08:44 AM
Thank you so much for taking time and explaining the concepts