Parquet column cannot be converted. Column: [Rainfall_Value], Expected: DoubleType, Found: INT64
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2023 06:41 AM
df.printSchema()
root
|-- Device_ID: string (nullable = true)
|-- Location: string (nullable = true)
|-- Latitude: double (nullable = true)
|-- Longitude: double (nullable = true)
|-- DateTime: timestamp (nullable = true)
|-- Rainfall_Value: double (nullable = true)
|-- year: integer (nullable = true)
|-- month: integer (nullable = true)
|-- day: integer (nullable = true)
|-- hour: integer (nullable = true)
|-- minute: integer (nullable = true)
df.write.partitionBy("year","month").mode("overwrite").parquet("/home/rainfall/parquet/rainfall.parquet")
org.apache.spark.SparkException: Job aborted due to stage failure: Task 29 in stage 675.0 failed 1 times, most recent failure: Lost task 29.0 in stage 675.0 (TID 5311) (ip-10-175-235-230.ap-southeast-1.compute.internal executor driver): com.databricks.sql.io.FileReadException: Error while reading file dbfs:REDACTED_LOCAL_PART@xyz**.com.sg/weather123-lakehouse/delta/2022-10-13.parquet/part-00003-tid-7527434428502281281-b966b165-5e61-4ba0-a6ca-cea51e5acdf2-3762-1-c000.snappy.parquet. Parquet column cannot be converted. Column: [Rainfall_Value], Expected: DoubleType, Found: INT64
Since the above schema already shows Rainfall_Value to be of DoubleType, why does it complain it found INT64 type? I am lost how to debug for this.
Thanks in advance.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2023 09:36 PM
Hi @THIAM HUAT TAN
Great to meet you, and thanks for your question!
Let's see if your peers in the community have an answer to your question. Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2023 04:00 PM
Yes, they have answered, thanks for checking.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2023 05:59 AM
Hi @THIAM HUAT TAN , The issue is because the schema defined for the column "Rainfall_Value" is of DoubleType and the values present in the data frame are of Integer type. This could be because of one or multiple values. Depending on the data, you need to update either of the one i.e. schema or the data.
