https://community.databricks.com/t5/data-engineering/issue-with-merge-into-for-first-batch/m-p/49959#M28664 <P>I have source data with multiple rows and columns, 1 of column is city.&nbsp;</P><P>I want to get unique city into other table by stream data from source table. So I trying to use merge into and foreachBatch with my merge function.&nbsp;&nbsp;My merge condition is : On source.city = target.city. But then I realized that, in first batch, I will always get duplicated data becauce there is no data on target and duplicated city on first batch of source.</P><P>How can I resolve this ? thank you for supporting</P> Fri, 27 Oct 2023 01:33:56 GMT Joe1912 2023-10-27T01:33:56Z https://community.databricks.com/t5/data-engineering/issue-with-merge-into-for-first-batch/m-p/49959#M28664 <P>I have source data with multiple rows and columns, 1 of column is city.&nbsp;</P><P>I want to get unique city into other table by stream data from source table. So I trying to use merge into and foreachBatch with my merge function.&nbsp;&nbsp;My merge condition is : On source.city = target.city. But then I realized that, in first batch, I will always get duplicated data becauce there is no data on target and duplicated city on first batch of source.</P><P>How can I resolve this ? thank you for supporting</P> Fri, 27 Oct 2023 01:33:56 GMT https://community.databricks.com/t5/data-engineering/issue-with-merge-into-for-first-batch/m-p/49959#M28664 Joe1912 2023-10-27T01:33:56Z