Walter_C
Databricks Employee
Databricks Employee

Can you please try with something like:

from pyspark.sql.types import StructType, StructField, StringType, DecimalType

# Define the schema with the appropriate precision and scale for decimal columns
schema = StructType([
    StructField("column1", StringType(), True),
    StructField("column2", StringType(), True),
    # Add other columns as needed
    StructField("litre_val", DecimalType(precision=10, scale=3), True),
    StructField("another_decimal_column", DecimalType(precision=10, scale=4), True)
    # Add other columns as needed
])

# Read the CSV file using the defined schema
spark_df = spark.read.format("csv") \
    .option("header", "true") \
    .schema(schema) \
    .load(f"s3://{s3_bucket}/{folder}/{filename}.csv")

# Display the DataFrame to verify the precision
spark_df.show()

In this example, replace "column1", "column2", etc., with the actual column names from your CSV file. The DecimalType(precision=10, scale=3) specifies that the litre_val column should be read as a decimal with a precision of 10 and a scale of 3. Adjust the precision and scale values as needed for your specific use case.