-werners-
Esteemed Contributor III

ok, so it is not registered as a UDF, which is good.

Looking at your code, nothing special there.  The reason why the performance is so bad is not clear to me.
It could be the table distribution (of source table or rates table), perhaps data skew?
If the rates table is small, it should be broadcasted so the join would go pretty fast.

I'd take a look into the spark UI and query plan what exactly is going on.  Because your code seems fine to me, as long as you don't apply a loop over records (I hope not)?