Databricks Data Type Conversion error

[Vibhor]

In databricks while writing data to curated layer, see error - Failed to execute user defined function (Double => decimal(38,18)). Does anyone know if faced such issue and how to resolve it.

[Anonymous]

Hi Vibhor,

Thanks for your question. Let's give it a while to see what the community comes up with. We'll circle back if we need to.

[-werners- · Accepted solution]

What happens if you explicitly cast it?

I remember having such issues with implicit casting when goin from spark 2.x to 3.x, but these were solved by using explicit casting (not round()).

[Vibhor]

When we multiply 2 numbers; the result size could go to the tune of 2x (num1 size + num2 size). Before we are explicitly casting multiplication of these 2, the result is bigger than to be casted data type. Its something like (double (x,y) can't be converted to decimal (x,y-1)). If we decrease the data type size of the values to me multiplied before multiplication we are getting incorrect value due to precision loss.

[Vibhor]

Scala 2.11 spark 2.4.5

[jose_gonzalez]

Hi @Vibhor Sethi​ ,

Could you please share more details on this issue? whats the error stack trace? could you share your code? are you writing to Delta or Parquet or which format?