topic How to get rid of "Command result size exceeds limit" in Data Engineering

How to get rid of "Command result size exceeds limit"

AmineHY — Wed, 11 Jan 2023 11:14:37 GMT

I am working on Databricks Notebook and trying to display a map using Floium and I keep getting this error

> Command result size exceeds limit: Exceeded 20971520 bytes (current = 20973510)

How can I get increase the memory limit?

I already reduced the size of my pandas dataframe.

Re: How to get rid of "Command result size exceeds limit"

daniel_sahal — Wed, 11 Jan 2023 11:46:53 GMT

Notebook has got a default 20MB limit for outputs. There's nothing you can do about that.

https://kb.databricks.com/en_US/jobs/job-cluster-limit-nb-output

Re: How to get rid of "Command result size exceeds limit"

AmineHY — Wed, 11 Jan 2023 12:00:08 GMT

Thanks, Daniel,

I have read this article, and I guess this problem occurs only while rendering the output, I'll try to save the result to disk in HTML format and visualize the results through the browser.

Have a good day.

Amine.

Re: How to get rid of "Command result size exceeds limit"

sher — Wed, 11 Jan 2023 17:27:36 GMT

you can try with another module like geopandas / pygal

Re: How to get rid of "Command result size exceeds limit"

labromb — Thu, 09 Mar 2023 09:26:22 GMT

Hi,

I have the same problem with keplergl, and the save to disk option, whilst helpful isn't super practical... So how does one plot large datasets in kepler?

Any thought welcome