โ12-07-2021 11:10 AM
In databricks while writing data to curated layer, see error - Failed to execute user defined function (Double => decimal(38,18)). Does anyone know if faced such issue and how to resolve it.
โ12-08-2021 12:47 AM
What happens if you explicitly cast it?
I remember having such issues with implicit casting when goin from spark 2.x to 3.x, but these were solved by using explicit casting (not round()).
โ12-07-2021 01:17 PM
Hi Vibhor,
Thanks for your question. Let's give it a while to see what the community comes up with. We'll circle back if we need to.
โ12-08-2021 12:47 AM
What happens if you explicitly cast it?
I remember having such issues with implicit casting when goin from spark 2.x to 3.x, but these were solved by using explicit casting (not round()).
โ12-08-2021 09:51 AM
When we multiply 2 numbers; the result size could go to the tune of 2x (num1 size + num2 size). Before we are explicitly casting multiplication of these 2, the result is bigger than to be casted data type. Its something like (double (x,y) can't be converted to decimal (x,y-1)). If we decrease the data type size of the values to me multiplied before multiplication we are getting incorrect value due to precision loss.
โ01-20-2022 11:43 PM
Hi @Vibhor Sethiโ , Can you tell me your dbr version?
โ02-08-2022 10:10 AM
Scala 2.11 spark 2.4.5
โ03-14-2022 05:15 PM
Hi @Vibhor Sethiโ ,
Could you please share more details on this issue? whats the error stack trace? could you share your code? are you writing to Delta or Parquet or which format?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.