@Vinay_M_R 

Thanks for your help.

I am afraid I didn't understand the reason why it is necessary.

This is because caching the DataFrame can cause it to lose any data skipping that can come from additional filters added on top of the cached DataFrame, 

---

Note that when df is cached, it is displayed immediately. 

• df.cache().count()
• df.display()

Then, a few more transformations are applied on "df" and the results are saved in "df_new" which is cached for display purposes:

• df_new.cache().count()
• df_new.display()

and the data that gets cached might not be updated if the table is accessed using a different identifier. 

---

Sorry I didn't understand this part that "if the table is accessed using a different identifier".

Therefore, it is recommended to assign the results of Spark transformations back to a SparkDataFrame variable, similar to how you might use common table expressions (CTEs), temporary views, or DataFrames in other systems.

---

It is done in the notebook. We assign result of transformation to a new DataFrame either caching is used or not.

Is there a reference in databricks documentation in this regard?