Snowflake Data Formatting Issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-22-2023 11:20 PM
I'm loading snowflake data to delta tables in databricks, few columns in snowflake data have datatype as Number (20,7) after loading to delta table it is taking as decimal (20,7), for example, if the value is 0.0000000 in snowflake then it is showing as 0E-7 in delta table, but i need it as 0.0000000 in delta table, I can't perform formatting function because there are many columns with data type as Number and with different decimal ranges, Please suggest how can I load it as 0.0000000
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-23-2023 01:25 AM
explicit casting seems like the way to go.
First try with one column, to see if that solves your issue.
If so, you can write a function that casts all decimal columns to a certain precision, something like this:
def convert_decimal_precision_scale(df, precision, scale):
# Iterate through each column and convert decimal columns to the specified precision and scale
for column in df.columns:
data_type = df.schema[column].dataType
if isinstance(data_type, DecimalType):
df = df.withColumn(column, col(column).cast(DecimalType(precision, scale)))
return df

