I have the case of deduplicating data source over specific business key using dense_rank function. Currently the data source does not have any duplicates, so the function should return 1 in all cases. The issue is that dense rank does not return proper integer, although data type is of integer:

• When filtering rank function equal to 1, it gives me "random" number of records. Most of dense_rank values with display values of 1 are getting dropped
• When filtering rank < 1.1 it gives me the same results as above
• When filtering rank > 0.9 it gives me the expected amount of rows
• When casting rank function to double and then filtering it as equal to 1, it gives me expected number of rows

 It happens on databricks runtime 13.1, so I am assuming spark 3.4 has this issue. It works with no problem with runtime 12.2