- 4939 Views
- 4 replies
- 5 kudos
Spark issue handling data from json when the schema DataType mismatch occurs
Hi,I have encountered a problem using spark, when creating a dataframe from a raw json source.I have defined an schema for my data and the problem is that when there is a mismatch between one of the column values and its defined schema, spark not onl...
- 4939 Views
- 4 replies
- 5 kudos
- 5 kudos
@Farzad Bonabi​ :Thank you for reporting this issue. It seems to be a known bug in Spark when dealing with malformed decimal values. When a decimal value in the input JSON data is not parseable by Spark, it sets not only that column to null but also ...
- 5 kudos
- 2616 Views
- 3 replies
- 6 kudos
Is there a function similar to split_part, json_extract_scalar?
I am using spark_sql version 3.2.1. Is there a function that can replacesplit_part,json_extract_scalarare not?
- 2616 Views
- 3 replies
- 6 kudos
- 6 kudos
pyspark.sql.functions.get_json_object(col, path)[source]Extracts json object from a json string based on json path specified, and returns json string of the extracted json object. It will return null if the input json string is invalid.​
- 6 kudos