- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-19-2026 10:06 AM - edited 03-19-2026 10:09 AM
Hi !
I’ve been exploring the new Managed Iceberg tables integration and noticed a potential discrepancy between the documentation and actual behavior regarding streaming/incremental workloads.
According to the official limitations, managed Iceberg tables do not currently support streaming or incremental batching. This seems particularly tied to declarative LakeFlow Streaming pipelines.
However, during my testing, I found a workaround:
Pre-declare the table in Unity Catalog (UC).
Execute the pipeline using pure Structured Streaming in Spark (decoupled from SQL Iceberg table creation syntax).
By following a specific syntax (similar to the one discussed in this community thread), the streaming process actually works.
My questions for the community and the Databricks team:
Is this "pure structure streaming" approach considered a supported workaround, or is it an unintended behavior (loophole) that might be restricted in future updates?
Given that the documentation explicitly lists streaming as a limitation, should we avoid this pattern for production workloads?
Thanks !
#Lakeflow #Declaratif pipeline #Streaming
- Labels:
-
Delta Lake
-
Spark