Can you perform a Merge into a Delta table from a stream?

[sajith_appukutt · Accepted solution]

You could MERGE into a delta table from a stream using foreachBatch. Here is a sample notebook

[Greg]

I am doing this exact thing with foreachBatch from a stream. However the Delta table I am merging into consumes a huge amount of space in relation to how much data is there due to the versions. Every MERGE execution will create new version. So if your stream is going to get lots of data that will trigger the MERGE, this may surprise you.

[Anonymous]

@Greg Moyer​ - Thank you. 🙂