topic Re: How can I throw an exception when a .json.gz file has multiple roots? in Data Engineering

How can I throw an exception when a .json.gz file has multiple roots?

amde99 — Thu, 11 Apr 2024 20:05:11 GMT

I have a situation where source files in .json.gz sometimes arrive with invalid syntax containing multiple roots separated by empty braces []. How can I detect this and thrown an exception? Currently the code runs and picks up only record set 1, and skips all the others without throwing any kind of exception.

Example of a bad source file (sorry for bad formatting, kept getting an invalid HTML error):

[

//...record set 1

"key": "value",
"key": "value",
"key": [
"value"
],
"key": "value",
"key": "value",
"key": "value"
},

][][

//... record set 2

][][][][

// ... record set 3

]

file read code:

df = spark.read \ .format("json") \ .load("<file path>.json.gz", multiLine=True) print(df.count())

Re: How can I throw an exception when a .json.gz file has multiple roots?

daniel_sahal — Fri, 19 Apr 2024 07:53:10 GMT

@amde99
Changing the mode to FAILFAST should be able to help you with throwing an exception.

https://spark.apache.org/docs/latest/sql-data-sources-json.html

Re: How can I throw an exception when a .json.gz file has multiple roots?

Lakshay — Fri, 19 Apr 2024 11:17:58 GMT

Schema validation should help here.