Need help with loading 11 TB data into spark dataframe using managed gcp databricks.

sh23 — Fri, 01 Jul 2022 08:18:52 GMT

I am using managed databricks on gcp. I have 11TB of data with 5B rows. Data from source is not partitioned. I'm having trouble loading the data into dataframe and do further data processing. I have tried couple of executors configuration , none of them seem to work. Can you guide me to best practise to load huge data into dataframe.

Data is in nested json format. Schema is not consistence across document. Source of data is mongoDB.

Things which I have tried already :

n1-standard-4 executors 20 - Job aborted after 2+ hours

n1-standard-8 executors 8 - Job aborted after 2 + hours

I know these are not best practises but I also tried setting the below spark config:

spark.executor.memory 0

spark.driver.memory 0

spark.driver.maxResultSize 0

I want to know what should be the right executor size, machine type , spark config to be used for my use case . Any suggestion which helps us save credits would be an added advantage. We plan to run data quality check for this data so we will be looking for reading the entire dataset.

Thanks in advance.

Re: Need help with loading 11 TB data into spark dataframe using managed gcp databricks.

jose_gonzalez — Wed, 20 Jul 2022 22:45:04 GMT

Hi @Shruti S,

Just a friendly follow-up. Are you getting any errors? please share the error stack trace so we can help you to narrow down the RCA for this issue.

topic Need help with loading 11 TB data into spark dataframe using managed gcp databricks. in Data Engineering

Need help with loading 11 TB data into spark dataframe using managed gcp databricks.

Re: Need help with loading 11 TB data into spark dataframe using managed gcp databricks.