topic Re: Snowflake Data Formatting Issue in Get Started Discussions

Snowflake Data Formatting Issue

rohithmalla — Wed, 23 Aug 2023 06:20:46 GMT

I'm loading snowflake data to delta tables in databricks, few columns in snowflake data have datatype as Number (20,7) after loading to delta table it is taking as decimal (20,7), for example, if the value is 0.0000000 in snowflake then it is showing as 0E-7 in delta table, but i need it as 0.0000000 in delta table, I can't perform formatting function because there are many columns with data type as Number and with different decimal ranges, Please suggest how can I load it as 0.0000000

Re: Snowflake Data Formatting Issue

-werners- — Wed, 23 Aug 2023 08:25:58 GMT

explicit casting seems like the way to go.
First try with one column, to see if that solves your issue.
If so, you can write a function that casts all decimal columns to a certain precision, something like this:

def convert_decimal_precision_scale(df, precision, scale): # Iterate through each column and convert decimal columns to the specified precision and scale for column in df.columns: data_type = df.schema[column].dataType if isinstance(data_type, DecimalType): df = df.withColumn(column, col(column).cast(DecimalType(precision, scale))) return df