Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-07-2016 01:52 AM
So you added "mergeSchema" so you would find the error immediately when some column types do not match? But how did you fix the actual problem of the same column having different types for different sets of same structure data? For example I have raw json data and there's a column which is supposed to be double type, but if by chance one set has only full integers, they are written in json without floating points and therefore spark doesn't know it has to be double and it reads it as long. Then when I write the same data as Parquet and later try to read this together with other sets, it fails