https://community.databricks.com/t5/data-engineering/how-can-process-3-5gb-gz-90gb-nested-json-and-convert-them-to/m-p/34054#M24848 <P>I have a total of 5000 files (Nested JSON ~ 3.5 GB). I have written a code which converts the json to Table in minutes (for JSON size till 1 GB) but when I am trying to process 3.5GB GZ json it is mostly getting failed because of Garbage collection. I have tried multiple clusters as well, still it is taking 18 minutes to read the file only and since it is a nested json it reads it only as a single record.</P><P>Please find the sample JSON Structure.</P><P><span class="lia-inline-image-display-wrapper" image-alt="Data frame structure"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1582i96FBF6A4B4D8BB8C/image-size/large?v=v2&amp;px=999" role="button" title="Data frame structure" alt="Data frame structure" /></span>Code snippet:</P><P><span class="lia-inline-image-display-wrapper" image-alt="Code"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1576i9468D0CE9962A6F0/image-size/large?v=v2&amp;px=999" role="button" title="Code" alt="Code" /></span>Reading Code:</P><P><span class="lia-inline-image-display-wrapper" image-alt="Reading Code"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1584iDEC8AC05168273BB/image-size/large?v=v2&amp;px=999" role="button" title="Reading Code" alt="Reading Code" /></span>I am looking a way first to process one 3.5 GZ file and after that my focus will be working on 5000 similar files. I am searching for a way which will be more optimized and cost effective. Currently I am using Azure Databricks but I am open you use any other alternate technology as well.</P> Mon, 22 Aug 2022 21:43:15 GMT sanchit_popli 2022-08-22T21:43:15Z https://community.databricks.com/t5/data-engineering/how-can-process-3-5gb-gz-90gb-nested-json-and-convert-them-to/m-p/34054#M24848 <P>I have a total of 5000 files (Nested JSON ~ 3.5 GB). I have written a code which converts the json to Table in minutes (for JSON size till 1 GB) but when I am trying to process 3.5GB GZ json it is mostly getting failed because of Garbage collection. I have tried multiple clusters as well, still it is taking 18 minutes to read the file only and since it is a nested json it reads it only as a single record.</P><P>Please find the sample JSON Structure.</P><P><span class="lia-inline-image-display-wrapper" image-alt="Data frame structure"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1582i96FBF6A4B4D8BB8C/image-size/large?v=v2&amp;px=999" role="button" title="Data frame structure" alt="Data frame structure" /></span>Code snippet:</P><P><span class="lia-inline-image-display-wrapper" image-alt="Code"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1576i9468D0CE9962A6F0/image-size/large?v=v2&amp;px=999" role="button" title="Code" alt="Code" /></span>Reading Code:</P><P><span class="lia-inline-image-display-wrapper" image-alt="Reading Code"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1584iDEC8AC05168273BB/image-size/large?v=v2&amp;px=999" role="button" title="Reading Code" alt="Reading Code" /></span>I am looking a way first to process one 3.5 GZ file and after that my focus will be working on 5000 similar files. I am searching for a way which will be more optimized and cost effective. Currently I am using Azure Databricks but I am open you use any other alternate technology as well.</P> Mon, 22 Aug 2022 21:43:15 GMT https://community.databricks.com/t5/data-engineering/how-can-process-3-5gb-gz-90gb-nested-json-and-convert-them-to/m-p/34054#M24848 sanchit_popli 2022-08-22T21:43:15Z