topic Re: Delta Live Tables - how do you merge multiple streaming datasets into a single one? in Data Engineering

Delta Live Tables - how do you merge multiple streaming datasets into a single one?

Ian — Thu, 30 Jun 2022 13:21:13 GMT

I need to create a dataset that is dependent on multiple streaming datasets. However, when I attempt to create the new single stream I am getting an error.

Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark;

Example:

create or refresh streaming live table silver_customer;

create temporary streaming live view customer_updates

with listOfCustomers as

(

select CustomerID

from

stream(live.raw_Customer)

union

select CustomerID

from

stream(live.raw_CustomerAddress)

)

select CustomerID

from listOfCustomers;

apply changes into

live.silver_customer

from

stream(live.customer_updates)

keys(CustomerID)

sequence by CustomerID

stored as scd type 1;

Re: Delta Live Tables - how do you merge multiple streaming datasets into a single one?

jose_gonzalez — Wed, 20 Jul 2022 22:50:55 GMT

Hi @Ian Corera,

Just a friendly follow-up. Do you still need help? please let us know if you do.

Re: Delta Live Tables - how do you merge multiple streaming datasets into a single one?

Ian — Thu, 21 Jul 2022 11:25:59 GMT

Hi Kaniz/Jose, I was able to resolve the issue. I used 'union all' to avoid aggregation on the stream and have it continue to write to the table in append mode.

This issue can be closed.