Great answer @Aman Sehgal​. I also received another answer from @Ryan Chynoweth​ I will paste here:

1) Have you seen anything like this before and if so, can you provide any insight on it?

Yes this does happen due to the lazy execution of spark and due to the dataset being distributed. Specifically with dropDuplicates it essentially keeps which ever row is returned first and that can change if the rows are on different nodes and/or more than 1 partition. There is a sort operation that happens so replication is not guaranteed with this function.

2) Is this purely a behavior of spark or the underlying frameworks, or could this be related to Databricks runtime or something we’re doing?

This behavior is not specific to Spark (or MPPs in general) but more related to the way dropDuplicates was created. Similarly, the df.first() function simply takes which ever row is returned first and in a distributed data.