The solution did not work for me. It was throwing an error stating: raise Py4JError( py4j.protocol.Py4JError: An error occurred while calling o434.readStream. Trace: py4j.Py4JException: Method readStream([class java. util.ArrayList]) does not exist.

An alternate solution which worked for me was:

import dlt

@dlt.table 
def final_table(): 
   table1 = spark.table("table_1") 
   table2 = spark.table("table_2")
   return table1.union(table2)

@dlt.table
def target():
    return dlt.apply_changes(
        target="target_table",
        source="final_table",
        keys=["key_column"],
        sequence_by="abc",
        stored_as_scd_type="1"
    )