topic Re: Databricks Data Type Conversion error in Data Engineering

Databricks Data Type Conversion error

Vibhor — Tue, 07 Dec 2021 19:10:38 GMT

In databricks while writing data to curated layer, see error - Failed to execute user defined function (Double => decimal(38,18)). Does anyone know if faced such issue and how to resolve it.

Re: Databricks Data Type Conversion error

Anonymous — Tue, 07 Dec 2021 21:17:46 GMT

Hi Vibhor,

Thanks for your question. Let's give it a while to see what the community comes up with. We'll circle back if we need to.

Re: Databricks Data Type Conversion error

-werners- — Wed, 08 Dec 2021 08:47:43 GMT

What happens if you explicitly cast it?

I remember having such issues with implicit casting when goin from spark 2.x to 3.x, but these were solved by using explicit casting (not round()).

Re: Databricks Data Type Conversion error

Vibhor — Wed, 08 Dec 2021 17:51:56 GMT

When we multiply 2 numbers; the result size could go to the tune of 2x (num1 size + num2 size). Before we are explicitly casting multiplication of these 2, the result is bigger than to be casted data type. Its something like (double (x,y) can't be converted to decimal (x,y-1)). If we decrease the data type size of the values to me multiplied before multiplication we are getting incorrect value due to precision loss.

Re: Databricks Data Type Conversion error

Vibhor — Tue, 08 Feb 2022 18:10:49 GMT

Scala 2.11 spark 2.4.5

Re: Databricks Data Type Conversion error

jose_gonzalez — Tue, 15 Mar 2022 00:15:40 GMT

Hi @Vibhor Sethi ,

Could you please share more details on this issue? whats the error stack trace? could you share your code? are you writing to Delta or Parquet or which format?