- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2023 06:07 PM
By default Databricks creates 2 volumes: one with 30GB and the other one with 150GB.
We have a lot of nodes in our pools and so a los of Terabytes of Volumes, but we are not making any use of them in the jobs. Is there any way to reduce the volumes? I couldn't found any documentation for reducing the default ebs volumes.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2023 10:22 PM
@pabloanzorenac
I don't think you'll be able to do that. Clusters are pre-defined and there's no way to change a configuration of each machine.
However even if you think that you're not using them in the jobs, Spark may actually do benefit by using them (ex. for caching).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-04-2023 08:05 AM
Yes, EBS vols are essential for shuffle spill for example. You are probably using them!
