Convert a Managed Table to Streaming Table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-29-2025 10:16 AM
Hi
I have applied transformations on a set of streaming tables and saved it as a managed table....
How can i change the Managed table to a Streaming table with minimal changes
Regards
ZD
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-29-2025 10:19 AM
Hi @zensardigital ,
Are you talking about Declarative pipelines (former DLT)? If so, I guess the simplest way is to just change the definition in code of DLT and re-run pipeline.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-29-2025 10:33 AM
I am just writing the dataframe to delta table.....Are you suggesting me to first define a STREAMING TABLE (using the DLT definition) and then save the dataframe into that table?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-29-2025 11:33 AM
Nope, at first I thought that you're using DLT. I think you're confusing things. Managed table means that your table is managed - so if you delete it you delete metadata along with data. You can also have an external table which when you execute delete statement will delete metadata from catalog (but the data won't be removed).
Back to your quesiton. So in your case, you probably saved you table using batch approach:
df_transformed.write.format("delta").saveAsTable("my_db.my_table")Then the easiest way convert it to streaming table would be to rewrite DataFrameWriter part:
df_transformed.writeStream
.format("delta")
.outputMode("append") # or "complete" depending on aggregation
.option("checkpointLocation", "/mnt/checkpoints/my_table_cp")
.toTable("my_db.my_table") # creates or writes to managed streaming tableBut keep in mind that not all operations are supported in spark streaming, so it's possible that if you have many complex transformation you won't be able to use streaming in that case.