topic Re: How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom do in Data Engineering

How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom docker images?

Alex_Persin — Thu, 28 Oct 2021 09:59:06 GMT

PyTorch uses shared memory to efficiently share tensors between its dataloader workers and its main process. However in a docker container the default size of the shared memory (a tmpfs file system mounted at /dev/shm) is 64MB, which is too small to use to share image tensor batches. This means that when using a custom docker image on a databricks cluster it is not possible to use PyTorch with multiple dataloaders. We can fix this by setting the `--shm-size` or `--ipc=host` args on `docker run` - how can this be set on a databricks cluster?

Note that this doesn't affect the default databricks runtime it looks like that is using the linux default of making half the physical RAM available to /dev/shm - 6.9GB on the Standard_DS3_v2 node I tested.

To reproduce: start a cluster using a custom docker image, run `df -h /dev/shm` in a notebook.

Thanks in advance!

Re: How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom docker images?

mstuder — Wed, 19 Jan 2022 15:25:46 GMT

Also interested in increasing shared memory for use with ray

Re: How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom docker images?

Alex_Persin — Thu, 20 Jan 2022 13:17:08 GMT

We spoke to DataBricks about this and they are working on it. At the beginning of the month they said it should be available on Jan 17th but I'm not sure of the status now, we ended up moving this workload off of the platform.

Re: How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom do

Hugh_Ku — Fri, 02 Aug 2024 09:55:13 GMT

Hey folks, any follow-up on this, or alternative solution? thanks

Re: How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom do

OxFF — Sat, 21 Sep 2024 10:12:50 GMT

Recently stumbled on this problem. It seems like it basically makes impossible usage of compute with custom docker images for any pytorch-based real life computer vision ML experiments. Which is unfortunate. +1 for requesting followup and possible alternative solutions! Thank you!

Re: How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom do

sashme — Thu, 16 Jan 2025 10:10:46 GMT

bump this one. interested in the topic too.
is there a known solution yet?

Re: How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom do

stevewb — Wed, 09 Apr 2025 13:54:01 GMT

Bump again... does anyone have a solution for this?