Hey @Cheuk Hin Christophe Poon​ 

You shared a screenshot of the whole cluster RAM usage, can you please look only on the driver metrics and check only his RAM usage ?

In my case the cluster metrics looked fine but when check only the driver i saw that my RAM got explode.

Besides this, I'm not sure if .pandas_api() collect all the data into driver, so you may want to check this.