topic Re: memory issues - databricks in Data Engineering

memory issues - databricks

pavanb — Tue, 05 Apr 2022 11:50:37 GMT

Hi All, All of a sudden in our Databricks dev environment, we are getting exceptions related to memory such as out of memory , result too large etc.

Also, the error message is not helping to identify the issue.

Can someone please guide on what would be the starting point to look into it.

I am getting this issue while reading a json file and dumping it into a dataframe.

Re: memory issues - databricks

Hubert-Dudek — Tue, 05 Apr 2022 12:20:25 GMT

OOM error is quite common, it means that probably your partitions are too large to fit into memory. Please analyze your SPARK UI - look for data spills, and skews and try to use smaller partitions or/and more shuffle partitions.

Re: memory issues - databricks

pavanb — Wed, 06 Apr 2022 09:43:20 GMT

Thanks for the response @Hubert Dudek .

if i run the same code in test environment , its getting successfully completed and in dev its giving out of memory issue. Also the configuration of test nand dev environment is exactly same.

Re: memory issues - databricks

jose_gonzalez — Fri, 29 Apr 2022 22:17:23 GMT

Hi @Pavan Bangad ,

Just a friendly follow-up. Did @Hubert Dudek 's response help you to resolve your issue? if yes, please select it as best answer. If not, please let us know, so we can continue helping you to find a solution.