Databricks Community

zzy · 02-01-2023

No matter GPU cluster of which size I create, cuda total capacity is always ~16 Gb. Does anyone know what is the issue?The code I use to get the total capacity:torch.cuda.get_device_properties(0).total_memory

zzy · 10-14-2022

I have a dataset about 5 million rows with 14 features and a binary target. I decided to train a pyspark random forest classifier on Databricks. The CPU cluster I created contains 2 c4.8xlarge workers (60GB, 36core) and 1 r4.xlarge (31GB, 4core) driv...

zzy · 08-10-2023

Hi Keval,Just to confirm that you are saying python multiprocessing will actually use working nodes instead of just the driver node. I thought worker nodes were only used under spark framework so python multiprocessing would only create all the proce...

zzy · 02-02-2023

Hi, the DBR version is 11.3 LTS ML. The instance types I created is g4dn.

Databricks Community

User Stats

User Activity

Why is pytorch cuda total memory not aligned with the memory size of GPU cluster I created?

Why is GPU accelerated node much slower than CPU node for training a random forest model on databricks?

Re: python multiprocessing and the Databricks Architecture - under the hood.

Re: Why is pytorch cuda total memory not aligned with the memory size of GPU cluster I created?