Hi everyone,

Has anyone run into this issue? I’m using the MongoDB Spark Connector on Databricks to expose data from Delta Lake to MongoDB. My workflow is:

1. overwrite the collection (very fast),

2. then create the indexes.

Occasionally, I’m seeing duplicates appear in MongoDB even though the Delta Lake source contains no duplicates. It looks like some Spark tasks sometimes fail and get retried, which leads to the same data being written twice, since there’s no uniqueness constraint at that moment.

Has anyone dealt with this behavior or found a reliable way to prevent duplicates during writes?