merge breaking persistance of dataframe

FabriceDeseyn — Wed, 06 Sep 2023 14:47:11 GMT

Hi all

In the minimal example below you can see that executing a merge statement trigger recomputation of a persisted dataframe. How does this happen?

from delta.tables import DeltaTable table_name = "hive_metastore.default.test_table" # initialize table data = [{"column1": 1}] df = spark.createDataFrame(data) df.write.saveAsTable(table_name, mode="overwrite") dt_dummy = DeltaTable.forName(spark, table_name) # Create new data new_data = [{"column1": 2}] df_new = spark.createDataFrame(new_data) # Identify differences df_diff = dt_dummy.toDF().join(df_new, "column1", "outer") df_diff = df_diff.persist() print(f"we now have {df_diff.count()} records in our data") # Execute merge print("executing merge") merge_results = ( dt_dummy.alias("deltatable") .merge(df_diff.alias("updates"), "updates.column1 != deltatable.column1") .whenMatchedUpdateAll() ).execute() # Identify differences after merge print(f"we now have {df_diff.count()} records in our persisted data") # --> recompute of persisted dataframe happened

This is run on DBR 13.2 and 13.3 LTS.

topic merge breaking persistance of dataframe in Data Engineering

merge breaking persistance of dataframe