At the Lakehouse, we don't enforce Primary Keys, which is why the deduplication strategy is so important. One of my favourites is using transformWithStateInPandas. Of course, it only makes sense in certain scenarios. See all five major strategies on my blog #databricks

https://databrickster.medium.com/deduplicating-data-on-the-databricks-lakehouse-5-ways-36a80987c716

https://www.sunnydata.ai/blog/databricks-deduplication-strategies-lakehouse