topic Re: how to create a nested(unflatten) json from flatten json in Data Engineering

how to create a nested(unflatten) json from flatten json

SailajaB — Wed, 19 Jan 2022 13:29:16 GMT

Hi ,

Is there any function in pyspark which can convert flatten json to nested json.

Ex : if we have attribute in flatten is like a_b_c : 23

then in unflatten it should be

{"a":{"b":{"c":23}}}

Thank you

Re: how to create a nested(unflatten) json from flatten json

cconnell — Wed, 19 Jan 2022 13:41:10 GMT

Your question is a little unclear. But see the lower section of this article, it might help...

https://towardsdatascience.com/json-in-databricks-and-pyspark-26437352f0e9

Re: how to create a nested(unflatten) json from flatten json

Hubert-Dudek — Thu, 20 Jan 2022 10:44:30 GMT

As @Chuck Connell said can you share more of your source json as that example is not json. Additionally flatten is usually to change something like {"status": {"A": 1,"B": 2}} to {"status.A": 1, "status.B": 2} which can be done easily with spark dataframes

Re: how to create a nested(unflatten) json from flatten json

SailajaB — Thu, 20 Jan 2022 10:54:09 GMT

Hi,

Thanks for your response.

We are looking for unflattening the flatten json.

If my json contains attributes like {"status_A":2,"status_B":3,"status_C_D":5} then we should unflatten the i/p flatten json one as below

{"status":{{"A":2,"B":3},{"C":{"D":5}}}

I hope the question is clear now

Thank you

Re: how to create a nested(unflatten) json from flatten json

jose_gonzalez — Wed, 09 Feb 2022 00:50:27 GMT

Hi @Sailaja B ,

I would like to share the following example notebook https://docs.databricks.com/_static/notebooks/transform-complex-data-types-scala.html it might be able to help you.

Thank you,

--Jose