Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2026 12:36 PM
Thanks @stbjelcevic ,
I am looking for a solution ....
=== Let's say, I have already had:
df1 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table1")
df2 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table2")
df1a = df1.select(....).transform(....)
df2a = df1.select(....).transform(....)
df = df1a.unionByName(df2a)
df.writeStream.format("delta").outputMode("append").option(
"checkpointLocation", my_checkpoint_path)
).trigger(availableNow=True).table(f"{silver_catalog}.{silver_schema}.my_silver_table")
=== Now, table3 is created by autoloader, and I read it as df3.
After some transformation, df3a has the same columns as the existing f"{silver_catalog}.{silver_schema}.my_silver_table".
df3 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table3")
df3a = df1.select(....).transform(....)
What would be the recommended solution to append the content of df3a to the existing f"{silver_catalog}.{silver_schema}.my_silver_table".