Delta Live Tables - how do you merge multiple streaming datasets into a single one?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-30-2022 06:21 AM
I need to create a dataset that is dependent on multiple streaming datasets. However, when I attempt to create the new single stream I am getting an error.
Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark;
Example:
create or refresh streaming live table silver_customer;
create temporary streaming live view customer_updates
as
with listOfCustomers as
(
select CustomerID
from
stream(live.raw_Customer)
union
select CustomerID
from
stream(live.raw_CustomerAddress)
)
select CustomerID
from listOfCustomers;
apply changes into
live.silver_customer
from
stream(live.customer_updates)
keys(CustomerID)
sequence by CustomerID
stored as scd type 1;
- Labels:
-
Delta Live Tables
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-20-2022 03:50 PM
Hi @Ian Corera,
Just a friendly follow-up. Do you still need help? please let us know if you do.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2022 04:25 AM
Hi Kaniz/Jose, I was able to resolve the issue. I used 'union all' to avoid aggregation on the stream and have it continue to write to the table in append mode.
This issue can be closed.