@Robert Pearce​ :

It is possible to achieve the desired behavior using apply_changes in Databricks Delta Lake. You can use the merge operation to merge data from your source into your target Delta table, and then use whenMatchedUpdate to update the id2 column to be equal to the id1 column in the source data.

Here's an example code snippet:

from delta.tables import *
 
# Define source and target tables
source_table = "bronze_table"
target_table = "silver_table"
 
# Read the source data as a DataFrame
source_df = spark.read.table(source_table)
 
# Create a DeltaTable object for the target table
target_delta_table = DeltaTable.forPath(spark, target_table)
 
# Merge data from the source table into the target table
target_delta_table.alias("t").merge(
    source_df.alias("s"),
    "s.id = t.id1"
).whenMatchedUpdate(
    set={"id2": "s.id"}
).whenNotMatchedInsert(
    values={"id1": "s.id", "id2": "s.id"}
).execute()

This code will merge the data from the bronze_table source table into the silver_table target table. The

whenMatchedUpdate clause updates the id2 column to be equal to the id1 column in the source data, and the whenNotMatchedInsert clause inserts new rows with the id1 and id2 columns set to the

id column in the source data.

If you have any issues or need further guidance, the Databricks documentation on Delta Lake's merge

operation and apply_changes function can be very helpful.