topic Is it ok to join on the decimal type fields? How does it affect the performance? in Data Engineering

Is it ok to join on the decimal type fields? How does it affect the performance?

DK03 — Wed, 30 Nov 2022 13:13:04 GMT

Re: Is it ok to join on the decimal type fields? How does it affect the performance?

-werners- — Wed, 30 Nov 2022 13:17:09 GMT

Sure it is ok.

The performance of a join is mainly determined by the shuffle itself, potential data skew and the type of join (broadcasthasjoin, shufflehashjoin etc)

Re: Is it ok to join on the decimal type fields? How does it affect the performance?

UmaMahesh1 — Wed, 30 Nov 2022 18:02:46 GMT

As @Werner Stinckens said, it would be ok. But generally decimal column joins are not recommended as other factors come into play like the precision, length etc..

.Also when you are joining in on decimal columns, be sure to check out the abs value of the difference between the two column values is nearly 0 basis on your requirement. e.g. 0.0000000001. This is because you don't want to mess up the join just because of a 1 in a billion difference or error which might be creeping in because of some transformations somewhere or data quality issues.

Do research well before doing joins on decimal columns.

Cheers.