cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

cannot convert Parquet type INT64 to Photon type double

auser85
New Contributor III

I am trying to read in files via the COPY INTO command but I am getting this error lately for a certain subset of the data;

`Error while reading file: Schema conversion error: cannot convert Parquet type INT64 to Photon type double`

These are my options; I have tried with a mixture of mergeSchema and overwriteSchema

What might I do to make this more reliable?

              FILEFORMAT = PARQUET
              FORMAT_OPTIONS ('overwriteSchema' = 'true')
              COPY_OPTIONS ('overwriteSchema' = 'true', 'overwrite' = 'true')

2 REPLIES 2

Aviral-Bhardwaj
Esteemed Contributor III

hey @Andrew Fogarty​ 

I also faced the same issue when I moved from the 7.3 LTS version to a higher runtime version so to mitigate this issue you can use the below cluster configuration

spark.sql.storeAssignmentPolicy LEGACY

spark.sql.parquet.binaryAsString true

spark.speculation false

spark.sql.legacy.timeParserPolicy LEGACY

For a detailed explanation of the above configuration please use this doc. this is really helpful to debug most of your errors

Spark configuration link- https://spark.apache.org/docs/latest/configuration.html

if you like my answer please upvote it.

Thanks

Aviral Bhardwaj

Thanks, will check it out!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.