11-03-2021 02:51 PM
We're in the process of migrating a large graph computation workload to nvidia RAPIDS + cuGraph for GPU acceleration.
The package isn't a part of the base runtime and it is available by conda package management only, so can't be installed via init script or package list.
We're a bit surprised it isn't part of the base; it's been on cloudera for over a year, for example.
We're going down the custom docker container route and are curious - anyone else doing the same thing? Have you built a dockerfile? If it's useful to others, we'll share our dockerfile and image in the container registry.
11-16-2021 08:21 AM
Hi @Brian Gue Databricks container services for GPU clusters is now in public preview. You can build your docker image to use it on a GPU cluster.
https://docs.databricks.com/clusters/gpu.html#databricks-container-services-on-gpu-clusters
You can find the GPU-supported docker image in Github.
https://github.com/databricks/containers/tree/master/ubuntu/gpu
11-03-2021 03:56 PM
Hi @Brian Gue! It's nice to meet you and thank you for your question. This sounds interesting and I look forward to seeing what our members come up with. If not, we will see what we can find for you.
11-16-2021 08:21 AM
Hi @Brian Gue Databricks container services for GPU clusters is now in public preview. You can build your docker image to use it on a GPU cluster.
https://docs.databricks.com/clusters/gpu.html#databricks-container-services-on-gpu-clusters
You can find the GPU-supported docker image in Github.
https://github.com/databricks/containers/tree/master/ubuntu/gpu
11-16-2021 11:32 AM
Thanks @Prabakar Ammeappin , we're looking at this. Strangely, the last commit removed the rapids libraries from the base cuda-images. We're adding them back in.
11-17-2021 09:31 AM
Thanks for the update @Brian Gue . Hope, adding the rapids libraries to the cuda-image helps to resolve your issue.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.