topic Ways to validate final Dataframe schema against JSON schema config file in Data Engineering

Ways to validate final Dataframe schema against JSON schema config file

SailajaB — Tue, 08 Feb 2022 13:07:00 GMT

Hi Team,

We have to validate transformed dataframe output schema with json schema config file.

Here is the scenario

Our input json schema and target json schema are different. Using Databricks we are doing the required schema changes. Now, we need to validate final dataframe schema against target JSON schema config file.

Note : JSON schema is very complex (it contains upto 7 level differences between input and output)

We tried with few python libraries but all are working fine for simple schemas without any issues.

We are looking for an approach like if we have any way to convert complex JSON schema config to sample json data (i.e target json schema config) so we can easily validate against final dataframe schema.

Thank you

Re: Ways to validate final Dataframe schema against JSON schema config file

weldermartins — Tue, 08 Feb 2022 21:56:38 GMT

Hello, for several levels you can use the functions explode(array(desired-level)).

/* A little bit for your understanding. */
df =   testDF\
.withColumn("company",explode(array("company")))\
.withColumn("employees",explode(array("company.employees")))\

Re: Ways to validate final Dataframe schema against JSON schema config file

SailajaB — Wed, 09 Feb 2022 04:31:54 GMT

Hi @welder martins ,

Thank you for your reply..

We are looking for ways to validate the output dataframe schema against the JSON schema config.

I hope above one will be useful if we need to flatten the nested json structure.

Re: Ways to validate final Dataframe schema against JSON schema config file

Anonymous — Wed, 09 Feb 2022 15:57:39 GMT

@Sailaja B - Hi! My name is Piper, and I'm a moderator for the community. Thanks for your question. Please let us know how things go. If @welder martins' response answers your question, would you be happy to come back and mark their answer as best? That will help other members find the answer more quickly.