Spark behavior while dealing with Actions & Transformations ?

Mradul07 — Thu, 27 Oct 2022 22:20:05 GMT

Hi,

My question is - what happens to the initial RDD after the action is performed on it. Does it disappear or stays in the memory or does it needs to be explicitly cached() if we want to use it again.

For eg : If I execute this in a sequence :

df_output= df_input.filter(...) -- > transformation_1

df_output.count() -- > Action_1

df_final = df_output.filter(...) --> Transformation_2

df_final.count() -- > Action_2

While executing Action_2, does Transformation_1 & 2 are both performed again or only the Transformation_2 (if this is the case where is the result of Transformation_1 stored meanwhile) ?

topic Spark behavior while dealing with Actions & Transformations ? in Data Engineering

Spark behavior while dealing with Actions & Transformations ?