Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2026 07:09 AM
Thanks again,
Option1: >> the easiest option would be to add a second streaming writer for df3a to the same existing f"{silver_catalog}.{silver_schema}.my_silver_table".
If I simply read the new bronze table3, then append only to the existing table, i.e.,
--------
df3 = spark.readStream.table(f"{bronze_catalog}.{bronze_schema}.table3")
df3a = df1.select(....).transform(....)
df3a.writeStream.format("delta").outputMode("append").option(
"checkpointLocation", my_checkpoint_path)
).trigger(availableNow=True).table(f"{silver_catalog}.{silver_schema}.my_silver_table")
--------
Spark will give errors like:
com.databricks.sql.transaction.tahoe.DeltaIllegalStateException: [DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE] The streaming query was reading from an unexpected Delta table (id = 'de2e002d-e6a7-452a-acd6-6843796fe00c').
Option2: >> doing a union with all dataframes from the different sources, then write once. i.e., the example for df1+df2 in my previous post.
I remember that I must create my_silver_table from scratch whenever unioning a new df. a.k.a, deleting the table created before adding the new df (of the new source) and its corresponding checkpoint. Otherwise, spark complains the streaming source is different.
Using the adding df3a as an example, the existing my_silver_table is created when the source has df1 & df2. After introducing df3, spark will not allow writing to my_silver_table unless I remove the cehckpoint. For consistency, remove the table itself too. Did I miss any important steps?