- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-28-2022 10:03 PM
If my cluster memory is 1GB for example and my data is 1TB how Spark will handle it?
If it is in memory computing how does it handles the data that is greater than the memory size ?
- Labels:
-
Cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-29-2022 05:12 AM
@Kaniz Fatma @Cedric Law Hing Ping
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-29-2022 05:33 AM
Hi @Abdullah Durrani,
Spark workers will spill the data on disk if the dataset is larger than the memory size.
I'd advise you to follow the best practices page https://docs.databricks.com/clusters/cluster-config-best-practices.html#cluster-sizing-consideration... to determine what cluster size you should configure for your use case.
