lizou
Contributor III
In response to Dicer

‎11-12-2022 10:42 AM

@Cheuk Hin Christophe Poon​ 

I am not sure how the library makes use of memory, maybe there is extra memory required to hold the data transformation process.

I also found an article about pandas won't go faster on a 20 node cluster.

https://learn.microsoft.com/en-us/azure/databricks/migration/single-node