Thank you very much for your response @ManojkMohan. By configuring the different options to avoid the retry problem, it may be possible to prevent the error from occurring. Additionally, by adding a dropDuplicates, would it be possible to avoid creating the staging table, or is it mandatory to create it in the SQL Server database

I still have doubts, because this is not a matter of duplicates in the DataFrame I’m trying to insert. The issue is that the DataFrame does not contain the record that the database or the transaction, when attempting the insert, is flagging.

For example, I print yesterday_df and the record exists in the database; I print today_df and the record also appears; but when I print inserts_df, that record is not present. Nevertheless, the transaction tries to insert it, even though the DataFrame I’m writing with the right join does not contain it. This leaves me a bit puzzled.

Ultimately, my question is: is it mandatory to create the staging table, or could proper configuration to avoid retries and other options I’m currently exploring allow us to avoid creating this table and then performing the MERGE?