cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

FarBo
by New Contributor III
  • 2280 Views
  • 4 replies
  • 5 kudos

Spark issue handling data from json when the schema DataType mismatch occurs

Hi,I have encountered a problem using spark, when creating a dataframe from a raw json source.I have defined an schema for my data and the problem is that when there is a mismatch between one of the column values and its defined schema, spark not onl...

  • 2280 Views
  • 4 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

@Farzad Bonabi​ :Thank you for reporting this issue. It seems to be a known bug in Spark when dealing with malformed decimal values. When a decimal value in the input JSON data is not parseable by Spark, it sets not only that column to null but also ...

  • 5 kudos
3 More Replies
Anonymous
by Not applicable
  • 491 Views
  • 0 replies
  • 0 kudos

Escaped quotes mess up table records

When table content is dumped from the RDBMS (e.g. Oracle), some column values may contain escaped double quotes (\") in the column values, which may cause the values from multiple columns to be concatenated into one value and result in corrupted reco...

  • 491 Views
  • 0 replies
  • 0 kudos
Labels