https://community.databricks.com/t5/data-engineering/garbage-collection-optimization/m-p/21805#M14897 <P>You can also tune the JVM's GC parameters directly, if you mean the pauses are too long. Set "spark.executor.extraJavaOptions", but it does require knowing a thing or two about how to tune for what performance goal.</P> Tue, 22 Jun 2021 16:06:59 GMT sean_owen 2021-06-22T16:06:59Z https://community.databricks.com/t5/data-engineering/garbage-collection-optimization/m-p/21803#M14895 <P>I have a case where garbage collection is taking much time and I want to optimize it for better performance</P> Tue, 22 Jun 2021 13:08:09 GMT https://community.databricks.com/t5/data-engineering/garbage-collection-optimization/m-p/21803#M14895 User16826994223 2021-06-22T13:08:09Z https://community.databricks.com/t5/data-engineering/garbage-collection-optimization/m-p/21804#M14896 <P>You can use smaller instances with less ram than the VMs with higher ram, However there will be a trade off if there are lots of shuffle involve in the operation, because more small memory vms will increase the shuffling operation time </P> Tue, 22 Jun 2021 13:12:26 GMT https://community.databricks.com/t5/data-engineering/garbage-collection-optimization/m-p/21804#M14896 User16826994223 2021-06-22T13:12:26Z https://community.databricks.com/t5/data-engineering/garbage-collection-optimization/m-p/21805#M14897 <P>You can also tune the JVM's GC parameters directly, if you mean the pauses are too long. Set "spark.executor.extraJavaOptions", but it does require knowing a thing or two about how to tune for what performance goal.</P> Tue, 22 Jun 2021 16:06:59 GMT https://community.databricks.com/t5/data-engineering/garbage-collection-optimization/m-p/21805#M14897 sean_owen 2021-06-22T16:06:59Z