- 4933 Views
- 4 replies
- 5 kudos
Spark issue handling data from json when the schema DataType mismatch occurs
Hi,I have encountered a problem using spark, when creating a dataframe from a raw json source.I have defined an schema for my data and the problem is that when there is a mismatch between one of the column values and its defined schema, spark not onl...
- 4933 Views
- 4 replies
- 5 kudos
- 5 kudos
@Farzad Bonabi​ :Thank you for reporting this issue. It seems to be a known bug in Spark when dealing with malformed decimal values. When a decimal value in the input JSON data is not parseable by Spark, it sets not only that column to null but also ...
- 5 kudos
- 780 Views
- 0 replies
- 0 kudos
Escaped quotes mess up table records
When table content is dumped from the RDBMS (e.g. Oracle), some column values may contain escaped double quotes (\") in the column values, which may cause the values from multiple columns to be concatenated into one value and result in corrupted reco...
- 780 Views
- 0 replies
- 0 kudos