topic Re: Getting OOM error while loading huge zipped CSV file to the databricks Hive_metasore table in Data Engineering

Getting OOM error while loading huge zipped CSV file to the databricks Hive_metasore table

gangs — Wed, 01 Mar 2023 12:23:15 GMT

Is any better way to load huge zipped CSV file to hive_metastore table ?????

Re: Getting OOM error while loading huge zipped CSV file to the databricks Hive_metasore table

daniel_sahal — Wed, 01 Mar 2023 13:41:41 GMT

@Ankit Gangwal

Scale up your cluster

Re: Getting OOM error while loading huge zipped CSV file to the databricks Hive_metasore table

Lakshay — Wed, 01 Mar 2023 14:03:29 GMT

Hi @Ankit Gangwal , The problem with the zip files is that they are not splittable and only use one core to process. It is better to change the compression format to snappy as it is splittable and will allow spark to distribute the workload over the cluster.

Ref link:- https://www.linkedin.com/pulse/apache-spark-optimizations-compression-deepak-rajak

Re: Getting OOM error while loading huge zipped CSV file to the databricks Hive_metasore table

Anonymous — Fri, 17 Mar 2023 04:53:35 GMT

Hi @Ankit Gangwal

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!