Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-02-2025 04:13 AM
Can you please try with something like:
from pyspark.sql.types import StructType, StructField, StringType, DecimalType
# Define the schema with the appropriate precision and scale for decimal columns
schema = StructType([
StructField("column1", StringType(), True),
StructField("column2", StringType(), True),
# Add other columns as needed
StructField("litre_val", DecimalType(precision=10, scale=3), True),
StructField("another_decimal_column", DecimalType(precision=10, scale=4), True)
# Add other columns as needed
])
# Read the CSV file using the defined schema
spark_df = spark.read.format("csv") \
.option("header", "true") \
.schema(schema) \
.load(f"s3://{s3_bucket}/{folder}/{filename}.csv")
# Display the DataFrame to verify the precision
spark_df.show()
In this example, replace "column1", "column2", etc., with the actual column names from your CSV file. The DecimalType(precision=10, scale=3) specifies that the litre_val column should be read as a decimal with a precision of 10 and a scale of 3. Adjust the precision and scale values as needed for your specific use case.