cdn_yyz_yul
Contributor III

Thanks @stbjelcevic ,

I am looking for a solution .... 

=== Let's say, I have already had:

df1 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table1")
df2 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table2")

df1a = df1.select(....).transform(....)
df2a = df1.select(....).transform(....)

df = df1a.unionByName(df2a)

df.writeStream.format("delta").outputMode("append").option(
    "checkpointLocation", my_checkpoint_path)
).trigger(availableNow=True).table(f"{silver_catalog}.{silver_schema}.my_silver_table")

=== Now, table3 is created by autoloader, and I read it as df3.
After some transformation, df3a has the same columns as the existing f"{silver_catalog}.{silver_schema}.my_silver_table".

df3 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table3")
df3a = df1.select(....).transform(....)

What would be the recommended solution to append the content of df3a to the existing f"{silver_catalog}.{silver_schema}.my_silver_table".