Parquet column cannot be converted. Column: [Rainfall_Value], Expected: DoubleType, Found: INT64

[THIAM_HUATTAN]

df.printSchema()

root

|-- Device_ID: string (nullable = true)

|-- Location: string (nullable = true)

|-- Latitude: double (nullable = true)

|-- Longitude: double (nullable = true)

|-- DateTime: timestamp (nullable = true)

|-- Rainfall_Value: double (nullable = true)

|-- year: integer (nullable = true)

|-- month: integer (nullable = true)

|-- day: integer (nullable = true)

|-- hour: integer (nullable = true)

|-- minute: integer (nullable = true)

df.write.partitionBy("year","month").mode("overwrite").parquet("/home/rainfall/parquet/rainfall.parquet")

org.apache.spark.SparkException: Job aborted due to stage failure: Task 29 in stage 675.0 failed 1 times, most recent failure: Lost task 29.0 in stage 675.0 (TID 5311) (ip-10-175-235-230.ap-southeast-1.compute.internal executor driver): com.databricks.sql.io.FileReadException: Error while reading file dbfs:REDACTED_LOCAL_PART@xyz**.com.sg/weather123-lakehouse/delta/2022-10-13.parquet/part-00003-tid-7527434428502281281-b966b165-5e61-4ba0-a6ca-cea51e5acdf2-3762-1-c000.snappy.parquet. Parquet column cannot be converted. Column: [Rainfall_Value], Expected: DoubleType, Found: INT64

Since the above schema already shows Rainfall_Value to be of DoubleType, why does it complain it found INT64 type? I am lost how to debug for this.

Thanks in advance.

[Anonymous]

Hi @THIAM HUAT TAN​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

[THIAM_HUATTAN]

Yes, they have answered, thanks for checking.

[Lakshay]

Hi @THIAM HUAT TAN​ , The issue is because the schema defined for the column "Rainfall_Value" is of DoubleType and the values present in the data frame are of Integer type. This could be because of one or multiple values. Depending on the data, you need to update either of the one i.e. schema or the data.