topic Re: When to persist and when to unpersist RDD in Spark in Data Engineering

When to persist and when to unpersist RDD in Spark

paourissi — Sun, 22 Nov 2015 21:03:31 GMT

Lets say i have the following:

<code>val dataset2 = dataset1.persist(StorageLevel.MEMORY_AND_DISK)

val dataset3 = dataset2.map(.....)1)

1)If you do a transformation on the dataset2 then you have to persist it and pass it to dataset3 and unpersist the previous or not?

2)I am trying to figure out when to persist and unpersist RDDs. With every new rdd that is created do i have to persist it?

3)In order for an unpersist to take place, an action must be following?(e.x otherrdd.count)

Thanks

Re: When to persist and when to unpersist RDD in Spark

Arun_KumarPT — Wed, 25 Nov 2015 06:10:50 GMT

It is well documented here : http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence

Re: When to persist and when to unpersist RDD in Spark

TimKellogg — Mon, 04 Nov 2019 19:14:24 GMT

This doesn't answer any of the questions asked. This question is about unpersisting a data frame. The linked docs only say that it can be done, but doesn't give any hints as to when it should be done. My worry is that unpersisting too soon will lead to zero cache benefits.

I assume that you should wait until the after last force evaluation, but it's not documented and it's hard to reason about given that cache/unpersist are mutating.