@lizou​ 

I also think so. Every time, when the memory use% reaches 50%+, Databricks will autoscale more workers.

3 days ago, I used pandas to transpose the data, but it seems super slow and failed due to GC. Yesterday, I tried to transpose the data only using PySpark.

I think Databricks needs to explain more about how they allocate memory and memory calculation methods.

btw, just now, I completed my data transposition.

Here is what I do:

1. Deploy one single 128GB memory VM to transpose the data for 17 hours despite the data size being far less than 128GB (actual data size is 2 GB, but the Databricks Live Metrics says it is 20GB. Maybe, I forgot to vacuum the delta tables? that's why it says 20GB?) and the theoretical complexity being O(RxN).

Here is my guess: Transposing a data matrix requires collecting all the data into one node, so deploying many smaller VMs may fail the data transposition. Transposing data can only be done in single-node mode.