Thanks Rishabh-Pandey for your reply!

So to clarify by giving some code example. Let's say this is a PROD code:

df1 = spark.read..... + transformations

df1.persist(StorageLevel.DISK_ONLY) or df1.cache()


df2 = df1.transformation...
df3 = df1.transformation...

df4 = df2.union(df3)

df4.write...

I noticed that some people are stating that:

1. If there's one action in the whole ETL then it's enough to persist/cache data. Like in the example above.
2. Other people are stating that if you persist/cache you need to right away after trigger some action (like in my previous question - take(5) for instance).

So who's right? Will Spark apply to persist/cache even if the action will be at the end of the bunch of transformations?

I understand that you need to call some action right after persisting/caching if you want to cache/persist some other dataframe on the way in a long chain of transformations and unpersist the previous one. Like here:

df1 = spark.read..... + transformations

df1.persist(StorageLevel.DISK_ONLY) or df1.cache()
df1.take(1) <<<<==== here

df2 = df1.transformation...
df3 = df1.transformation...

df4 = df2.union(df3)
df4.persist(StorageLevel.DISK_ONLY)
df1.unpersist()

df4.write...

But if that's not the case, then what is the purpose? Does it make any sense?