What is the best approach to display DataFrame without re-executing the logic each time we display?

Mado
Valued Contributor II

Hi,

I have a DataFrame and different transformations are applied on the DataFrame. I want to display DataFrame after several transformations to check the results. 

 

However, according to the Reference, every time I try to display results, it runs the execution plan again. A solution has been proposed in the reference by saving the DataFrame and then loading it. However, this solution cannot be applied to the platform I am working on. 

 

Is there any other solution to display results a few times in a notebook without re-executing the logic?

 

Can I use .cache() for this purpose as below:

  • df.cache().count()
  • df.display()

And since the name of DataFrame will change in the next lines, I repeat it like below:

  • df_new.cache().count()
  • df_new.display()