topic DLT streaming table and LEFT JOIN in Data Engineering

DLT streaming table and LEFT JOIN

Daba — Tue, 07 Jun 2022 16:16:24 GMT

I'm trying to build gold level streaming live table based on two streaming silver live tables with left join.

This attempt fails with the next error:

"Append mode error: Stream-stream LeftOuter join between two streaming DataFrame/Datasets is not supported without a watermark in the join keys, or a watermark on the nullable side and an appropriate range condition".

My question is how I can implement the watermark with SQL interface? Where can I find examples?

Thanks.

Re: DLT streaming table and LEFT JOIN

Daba — Sun, 12 Jun 2022 08:02:58 GMT

Thanks Fatma,

I do understand the need for watermarks, but I'm just wondering if this supported by SQL syntax?

Re: DLT streaming table and LEFT JOIN

TomRenish — Fri, 24 Feb 2023 19:45:15 GMT

I'm wrestling with similar questions. This article makes it pretty clear that you'll need to use scala/python/java/r: streaming writes

Re: DLT streaming table and LEFT JOIN

NateAnth — Fri, 24 Feb 2023 22:18:08 GMT

It is now!

https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-watermark.html