https://community.databricks.com/t5/data-engineering/how-spark-will-handles-1tb-data-if-it-has-cluster-of-1gb-memory/m-p/15841#M10114 <P>@Kaniz Fatma​&nbsp;@Cedric Law Hing Ping​&nbsp;</P> Wed, 29 Jun 2022 12:12:19 GMT abd 2022-06-29T12:12:19Z https://community.databricks.com/t5/data-engineering/how-spark-will-handles-1tb-data-if-it-has-cluster-of-1gb-memory/m-p/15840#M10113 <P>If my cluster memory is 1GB for example and my data is 1TB how Spark will handle it?</P><P>If it is in memory computing how does it handles the data that is greater than the memory size ?</P> Wed, 29 Jun 2022 05:03:10 GMT https://community.databricks.com/t5/data-engineering/how-spark-will-handles-1tb-data-if-it-has-cluster-of-1gb-memory/m-p/15840#M10113 abd 2022-06-29T05:03:10Z https://community.databricks.com/t5/data-engineering/how-spark-will-handles-1tb-data-if-it-has-cluster-of-1gb-memory/m-p/15841#M10114 <P>@Kaniz Fatma​&nbsp;@Cedric Law Hing Ping​&nbsp;</P> Wed, 29 Jun 2022 12:12:19 GMT https://community.databricks.com/t5/data-engineering/how-spark-will-handles-1tb-data-if-it-has-cluster-of-1gb-memory/m-p/15841#M10114 abd 2022-06-29T12:12:19Z https://community.databricks.com/t5/data-engineering/how-spark-will-handles-1tb-data-if-it-has-cluster-of-1gb-memory/m-p/15842#M10115 <P>Hi @Abdullah Durrani​,</P><P></P><P>Spark workers will spill the data on disk if the dataset is larger than the memory size.</P><P></P><P>I'd advise you to follow the best practices page <A href="https://docs.databricks.com/clusters/cluster-config-best-practices.html#cluster-sizing-considerations" target="test_blank">https://docs.databricks.com/clusters/cluster-config-best-practices.html#cluster-sizing-considerations</A> to determine what cluster size you should configure for your use case.</P> Wed, 29 Jun 2022 12:33:25 GMT https://community.databricks.com/t5/data-engineering/how-spark-will-handles-1tb-data-if-it-has-cluster-of-1gb-memory/m-p/15842#M10115 Cedric 2022-06-29T12:33:25Z