dataframe - cast string to decimal when encountering zeros returns OE-16

shan_chandra
Databricks Employee
Databricks Employee

The user is trying to cast string to decimal when encountering zeros. The cast function displays the  '0' as '0E-16'. could you please let us know your thoughts on whether 0s can be displayed as 0s?

from pyspark.sql import functions as F
df = spark.sql("select cast('0' AS decimal(38,16)) as decimal_number union all select cast('1.0000123400000' AS decimal(38,16))")
df2 = df.withColumn("string_column", F.expr("format_number(decimal_number, '0.######################')"))
display(df2)  

Screen Shot 2022-03-09 at 12.13.11 PM