Can you perform a Merge into a Delta table from a stream?

User16826992666
Databricks Employee
Databricks Employee
 

sajith_appukutt
Databricks Employee
Databricks Employee

You could MERGE into a delta table from a stream using foreachBatch. Here is a sample notebook

View solution in original post

Greg
New Contributor III

I am doing this exact thing with foreachBatch from a stream. However the Delta table I am merging into consumes a huge amount of space in relation to how much data is there due to the versions. Every MERGE execution will create new version. So if your stream is going to get lots of data that will trigger the MERGE, this may surprise you.

Anonymous
Not applicable

@Greg Moyer​ - Thank you. 🙂