I think deploying a larger VM with more memory may work. In the ML case, deploying one big VM is better. But Spark workers will also spill the data on disk if the dataset is larger than the memory size. So why is there a memory problem? Is there any specific operation that the spark workers don't split data on disk?

https://community.databricks.com/s/question/0D58Y00008tIaMySAK/how-spark-will-handles-1tb-data-if-it...