How can the shared memory size (/dev/shm) be increased on databricks worker nodes with custom docker images?

[Alex_Persin]

PyTorch uses shared memory to efficiently share tensors between its dataloader workers and its main process. However in a docker container the default size of the shared memory (a tmpfs file system mounted at /dev/shm) is 64MB, which is too small to use to share image tensor batches. This means that when using a custom docker image on a databricks cluster it is not possible to use PyTorch with multiple dataloaders. We can fix this by setting the `--shm-size` or `--ipc=host` args on `docker run` - how can this be set on a databricks cluster?

Note that this doesn't affect the default databricks runtime it looks like that is using the linux default of making half the physical RAM available to /dev/shm - 6.9GB on the Standard_DS3_v2 node I tested.

To reproduce: start a cluster using a custom docker image, run `df -h /dev/shm` in a notebook.

Thanks in advance!

[mstuder]

Also interested in increasing shared memory for use with ray

[Alex_Persin]

We spoke to DataBricks about this and they are working on it. At the beginning of the month they said it should be available on Jan 17th but I'm not sure of the status now, we ended up moving this workload off of the platform.

[Hugh_Ku]

Hey folks, any follow-up on this, or alternative solution? thanks

[OxFF]

Recently stumbled on this problem. It seems like it basically makes impossible usage of compute with custom docker images for any pytorch-based real life computer vision ML experiments. Which is unfortunate. +1 for requesting followup and possible alternative solutions! Thank you!

[sashme]

bump this one. interested in the topic too.
is there a known solution yet?