topic Re: Can you perform a Merge into a Delta table from a stream? in Data Engineering

Can you perform a Merge into a Delta table from a stream?

User16826992666 — Wed, 16 Jun 2021 22:39:45 GMT

Re: Can you perform a Merge into a Delta table from a stream?

sajith_appukutt — Fri, 18 Jun 2021 07:21:30 GMT

You could MERGE into a delta table from a stream using foreachBatch. Here is a sample notebook

Re: Can you perform a Merge into a Delta table from a stream?

Greg — Tue, 28 Sep 2021 17:16:01 GMT

I am doing this exact thing with foreachBatch from a stream. However the Delta table I am merging into consumes a huge amount of space in relation to how much data is there due to the versions. Every MERGE execution will create new version. So if your stream is going to get lots of data that will trigger the MERGE, this may surprise you.

Re: Can you perform a Merge into a Delta table from a stream?

Anonymous — Thu, 30 Sep 2021 17:42:55 GMT

@Greg Moyer - Thank you. 🙂