topic Re: Can't run Display command. Any idea why I got this error message? How can I fix this? The nodes are at 28gb ram and memory usage : 162.9+mb so it shouldn't be memory issue :(. Please help! Thank you in advanced! in Data Engineering

Can't run Display command. Any idea why I got this error message? How can I fix this? The nodes are at 28gb ram and memory usage : 162.9+mb so it shouldn't be memory issue :(. Please help! Thank you in advanced!

sm1 — Mon, 13 Feb 2023 21:32:37 GMT

Re: Can't run Display command. Any idea why I got this error message? How can I fix this? The nodes are at 28gb ram and memory usage : 162.9+mb so it shouldn't be memory issue :(. Please help! Thank you in advanced!

Ajay-Pandey — Tue, 14 Feb 2023 06:09:27 GMT

Hi @Suky Muliadikara ,

The error is OutofMemory that means when you're calling display it's actually executing all the transformation that's causing OOM exception please increase your memory for driver and worker then you can use display ().

Re: Can't run Display command. Any idea why I got this error message? How can I fix this? The nodes are at 28gb ram and memory usage : 162.9+mb so it shouldn't be memory issue :(. Please help! Thank you in advanced!

-werners- — Tue, 14 Feb 2023 13:11:23 GMT

spark is lazy evaluated. this means actual compute only starts at an action (like display).

without seeing the rest of the code it is impossible to see what the issue is.

According to the error message, there is an OOM error, most probably caused by some code somewhere.

Increasing memory is an option but I suggest reviewing why you use so much memory.

Re: Can't run Display command. Any idea why I got this error message? How can I fix this? The nodes are at 28gb ram and memory usage : 162.9+mb so it shouldn't be memory issue :(. Please help! Thank you in advanced!

Lakshay — Wed, 15 Feb 2023 13:09:35 GMT

Hi @Suky Muliadikara , Could you please share more details like DBR version, number of nodes etc? Also, if you could share a screenshot of spark sql plan?

Re: Can't run Display command. Any idea why I got this error message? How can I fix this? The nodes are at 28gb ram and memory usage : 162.9+mb so it shouldn't be memory issue :(. Please help! Thank you in advanced!

Anonymous — Fri, 17 Feb 2023 05:47:20 GMT

Hi @Suky Muliadikara

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Can't run Display command. Any idea why I got this error message? How can I fix this? The nodes are at 28gb ram and memory usage : 162.9+mb so it shouldn't be memory issue :(. Please help! Thank you in advanced!

jose_gonzalez — Thu, 23 Feb 2023 22:09:25 GMT

Hi @Suky Muliadikara,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.