merge breaking persistance of dataframe

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi all

In the minimal example below you can see that executing a merge statement trigger recomputation of a persisted dataframe. How does this happen?

from delta.tables import DeltaTable

table_name = "hive_metastore.default.test_table"

# initialize table
data = [{"column1": 1}]
df = spark.createDataFrame(data)
df.write.saveAsTable(table_name, mode="overwrite")
dt_dummy = DeltaTable.forName(spark, table_name)

# Create new data
new_data = [{"column1": 2}]
df_new = spark.createDataFrame(new_data)

# Identify differences
df_diff = dt_dummy.toDF().join(df_new, "column1", "outer")
df_diff = df_diff.persist()
print(f"we now have {df_diff.count()} records in our data")

# Execute merge
print("executing merge")
merge_results = (
    dt_dummy.alias("deltatable")
    .merge(df_diff.alias("updates"), "updates.column1 != deltatable.column1")
    .whenMatchedUpdateAll()
).execute()
# Identify differences after merge
print(f"we now have {df_diff.count()} records in our persisted data")

# --> recompute of persisted dataframe happened