cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
cancel
Showing results for 
Search instead for 
Did you mean: 

Pytorch DDP on Databricks

Marco98
New Contributor II

Hello!

I am currently trying to use Pytorch Lightning inside Databricks and I am currently using a cluster with 2 gpus. Whenever I try to train my Transformer model with 1 gpu in DP strategy everything works fine, but when I try to use both the 2 gpus with a DDP strategy I get the following error:

MisconfigurationException: `Trainer(strategy='ddp_spawn')` or `Trainer(accelerator='ddp_spawn')` is not compatible with an interactive environment. Run your code as a script, or choose one of the compatible strategies: Trainer(strategy=None|dp|tpu_spawn). In case you are spawning processes yourself, make sure to include the Trainer creation inside the worker function.

1 REPLY 1

Chakra03
New Contributor II

Hi @Marco Capusso​ , I am facing the similar issue could you find some fix. It would be great if you share some details around it.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.