- 5387 Views
- 1 replies
- 0 kudos
How to merge parquets with different column types
ProblemI have a directory in S3 with a bunch of data files, like "data-20221101.parquet". They all have the same columns: timestamp, reading_a, reading_b, reading_c. In the earlier files, the readings are floats, but in the later ones they are double...
- 5387 Views
- 1 replies
- 0 kudos
- 0 kudos
1) Can you let us know what was the error message when you don't set the schema & use mergeSchema2) What happens when you define schema (with FloatType) & use mergeSchema ? what error message do you get ?
- 0 kudos
- 4698 Views
- 5 replies
- 2 kudos
Resolved! Databricks Data Type Conversion error
In databricks while writing data to curated layer, see error - Failed to execute user defined function (Double => decimal(38,18)). Does anyone know if faced such issue and how to resolve it.
- 4698 Views
- 5 replies
- 2 kudos
- 2 kudos
What happens if you explicitly cast it?I remember having such issues with implicit casting when goin from spark 2.x to 3.x, but these were solved by using explicit casting (not round()).
- 2 kudos