topic Re: Analyzing 23 GB JSON file in Data Engineering

Analyzing 23 GB JSON file

jayallenmn — Thu, 21 Jul 2022 04:30:13 GMT

Hey all,

We're trying to analyze the data in a 23 GB JSON file. We're using the basic starter cluster - one node, 2 cpu x 8GB.

We can read the JSON file into a spark dataframe and print out the schema but if we try and do any operations that won't cause a collect (take, filter), the driver fails with "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached."

The JSON file is multiline and it sounds like the entire thing will have to be read into memory on a node - so we need a cluster of larger nodes . What size cluster would you guys recommend? We were looking at a cluster of 3 8 x 32s - do you think that would work?

Jay

Re: Analyzing 23 GB JSON file

Prabakar — Thu, 21 Jul 2022 13:45:14 GMT

Hi @Jay Allen you can refer to the cluster sizing doc.

Re: Analyzing 23 GB JSON file

jayallenmn — Thu, 21 Jul 2022 19:50:41 GMT

Thanks Prabakar! We have 12 days left in our trial - we'd have to pay for the AWS VMs but would the databricks piece be free during the trial with the new, bigger cluster?