I want to thank you for this, because i was going crazy that my dataframe suddenly became empty.I used this method to easily delete duplicates from a Unity Catalog table, by de-duplicating with pypsark only the filtered dataframe with duplicates, del...
This is basically wiping and rewriting the whole table. Obviously it's a very easy solution, but very expensive.There's a reason why the "usual" solutions are very complex, because they only target the duplicated rows.
