Dataframe getting updated after creating temporary view

[Einsatz]

I'm observing different behavior between Databricks Runtime versions when working with DataFrames and temporary views, and would appreciate any clarification.

In both environments, I performed the following steps in a notebook (each connected to its own cluster):

1. transformed_df.createOrReplaceTempView("source_vw")
2. {few transformations}
3. transformed_df.count()
4. transformed_df.createOrReplaceTempView("source_vw")
5. transformed_df.count()

The transformations (step 2) involve modifying the same DataFrame (transformed_df), and at certain points, a temporary view is created using the same name (transformed_df).

In DBR 15.4, reusing the same name for the DataFrame and temporary view doesn't appear to overwrite the dataframes or temporary view as expected.

Behavior:

• In DBR 13.3 LTS, the results from Step 3 and Step 5 are identical (which is expected).
• In DBR 15.4 LTS, the result from Step 5 is different than Step 3.

To clarify, there were no writes or modifications to the underlying Delta table between the two counts. I’m just calling createOrReplaceTempView() a second time.

Attached images for reference.

Thanks in advance for any insight or references to release notes that could help explain this behavior.

[nikhilj0421]

Hi @Einsatz, this is expected in DBR version 14.3 and above since we don't have a Spark context. This is happening due to the cache invalidation. 

To resolve the issue, please go with the dynamic name for the view every time.