topic Re: Schema Parsing issue when datatype of source field is mapped incorrect in Data Engineering

Schema Parsing issue when datatype of source field is mapped incorrect

MattM — Mon, 17 Jan 2022 23:04:29 GMT

I have complex json file which has massive struct column. We regularly have issues when we try to parse this json file by forming our case class to extract the fields from schema. With this approach the issue we are facing is that if one data type of field within the case class is incorrect, the rest of the following fields in that class do not populate in the target. Hope the problem makes sense.

Is there any alternate way? One I can think of is to extract all the fields as string from json file and then do the data type conversion. This adds an extra step. A better solution is appreciated. Thanks.

Re: Schema Parsing issue when datatype of source field is mapped incorrect

Hubert-Dudek — Tue, 18 Jan 2022 21:11:46 GMT

I think solution for your problem is use auto loader stream to read data as it support schema hints. If you don't want to use it as stream is enough to specify there trigger once (so once all json are loaded it will finish a job).

Here is about loading json:

https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-json.html

then you can specify schema hints:

https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-schema.html#schema-hints

additionally you can experiment with different schema evolution options for stream

Re: Schema Parsing issue when datatype of source field is mapped incorrect

MattM — Wed, 19 Jan 2022 01:19:47 GMT

Thanks Hubert! I did have Autoloader as one of the solution and I think this is a viable option to make sure I do not have schema parsing issues.

Re: Schema Parsing issue when datatype of source field is mapped incorrect

Anonymous — Wed, 19 Jan 2022 16:43:13 GMT

Hey there, @Matt M - If @Hubert Dudek's response solved the issue, would you be happy to mark his answer as best? It helps other members find the solution more quickly.

Re: Schema Parsing issue when datatype of source field is mapped incorrect

MattM — Thu, 20 Jan 2022 15:23:11 GMT

Yes, thanks.