cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

cannot convert Parquet type INT64 to Photon type double

auser85
New Contributor III

I am trying to read in files via the COPY INTO command but I am getting this error lately for a certain subset of the data;

`Error while reading file: Schema conversion error: cannot convert Parquet type INT64 to Photon type double`

These are my options; I have tried with a mixture of mergeSchema and overwriteSchema

What might I do to make this more reliable?

              FILEFORMAT = PARQUET
              FORMAT_OPTIONS ('overwriteSchema' = 'true')
              COPY_OPTIONS ('overwriteSchema' = 'true', 'overwrite' = 'true')

2 REPLIES 2

Aviral-Bhardwaj
Esteemed Contributor III

hey @Andrew Fogarty​ 

I also faced the same issue when I moved from the 7.3 LTS version to a higher runtime version so to mitigate this issue you can use the below cluster configuration

spark.sql.storeAssignmentPolicy LEGACY

spark.sql.parquet.binaryAsString true

spark.speculation false

spark.sql.legacy.timeParserPolicy LEGACY

For a detailed explanation of the above configuration please use this doc. this is really helpful to debug most of your errors

Spark configuration link- https://spark.apache.org/docs/latest/configuration.html

if you like my answer please upvote it.

Thanks

Aviral Bhardwaj

Thanks, will check it out!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!