topic Re: Why driver memory is capped in Data Engineering

Why driver memory is capped

MikeGo — Thu, 26 Sep 2024 06:37:46 GMT

Hi team

We are using a job cluster to run spark with MERGE. Somehow it needs a lot driver memory. We allocate 128G+16core node for driver, and specify spark.driver.memory=96000m. We can see it is 96000m from env table of spark UI. The config is like:

"spark.driver.memory": "96000m",
"spark.memory.offHeap.size": "11872m",
"spark.executor.memory": "86000m",

however from metrics of the cluster, the driver memory is capped below 48G. How to make driver to fully use the memory?

Re: Why driver memory is capped

Kannathasan — Thu, 26 Sep 2024 13:27:58 GMT

Could you please try increase the partition the Dataframe by doing repartition() before you merge.

Re: Why driver memory is capped

MikeGo — Fri, 27 Sep 2024 01:08:02 GMT

Thanks for response. We are doubt why driver memory cannot be fully used (only 48G out of 128G is used for driver). Is this related with repartition?