When to persist and when to unpersist RDD in Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-22-2015 01:03 PM
Lets say i have the following:
<code>val dataset2 = dataset1.persist(StorageLevel.MEMORY_AND_DISK)val dataset3 = dataset2.map(.....)1)
1)If you do a transformation on the dataset2 then you have to persist it and pass it to dataset3 and unpersist the previous or not?
2)I am trying to figure out when to persist and unpersist RDDs. With every new rdd that is created do i have to persist it?
3)In order for an unpersist to take place, an action must be following?(e.x otherrdd.count)
Thanks
- Labels:
-
Scala
-
Spark scala
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-24-2015 10:10 PM
It is well documented here : http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-04-2019 11:14 AM
This doesn't answer any of the questions asked. This question is about unpersisting a data frame. The linked docs only say that it can be done, but doesn't give any hints as to when it should be done. My worry is that unpersisting too soon will lead to zero cache benefits.
I assume that you should wait until the after last force evaluation, but it's not documented and it's hard to reason about given that cache/unpersist are mutating.
