topic Re: Reduce EBS Default Volumes in Get Started Discussions

Reduce EBS Default Volumes

pabloanzorenac — Tue, 01 Aug 2023 01:07:29 GMT

By default Databricks creates 2 volumes: one with 30GB and the other one with 150GB.
We have a lot of nodes in our pools and so a los of Terabytes of Volumes, but we are not making any use of them in the jobs. Is there any way to reduce the volumes? I couldn't found any documentation for reducing the default ebs volumes.
Thanks

Re: Reduce EBS Default Volumes

daniel_sahal — Tue, 01 Aug 2023 05:22:05 GMT

@pabloanzorenac
I don't think you'll be able to do that. Clusters are pre-defined and there's no way to change a configuration of each machine.

However even if you think that you're not using them in the jobs, Spark may actually do benefit by using them (ex. for caching).

Re: Reduce EBS Default Volumes

sean_owen — Fri, 04 Aug 2023 15:05:01 GMT

Yes, EBS vols are essential for shuffle spill for example. You are probably using them!