topic Re: Anyone using RAPIDS and cuGraph on a current runtime? in Data Engineering

Anyone using RAPIDS and cuGraph on a current runtime?

Anonymous — Wed, 03 Nov 2021 21:51:11 GMT

We're in the process of migrating a large graph computation workload to nvidia RAPIDS + cuGraph for GPU acceleration.

The package isn't a part of the base runtime and it is available by conda package management only, so can't be installed via init script or package list.

We're a bit surprised it isn't part of the base; it's been on cloudera for over a year, for example.

We're going down the custom docker container route and are curious - anyone else doing the same thing? Have you built a dockerfile? If it's useful to others, we'll share our dockerfile and image in the container registry.

Re: Anyone using RAPIDS and cuGraph on a current runtime?

Anonymous — Wed, 03 Nov 2021 22:56:39 GMT

Hi @Brian Gue! It's nice to meet you and thank you for your question. This sounds interesting and I look forward to seeing what our members come up with. If not, we will see what we can find for you.

Re: Anyone using RAPIDS and cuGraph on a current runtime?

Prabakar — Tue, 16 Nov 2021 16:21:03 GMT

Hi @Brian Gue Databricks container services for GPU clusters is now in public preview. You can build your docker image to use it on a GPU cluster.

https://docs.databricks.com/clusters/gpu.html#databricks-container-services-on-gpu-clusters

You can find the GPU-supported docker image in Github.

https://github.com/databricks/containers/tree/master/ubuntu/gpu

Re: Anyone using RAPIDS and cuGraph on a current runtime?

Anonymous — Tue, 16 Nov 2021 19:32:00 GMT

Thanks @Prabakar Ammeappin , we're looking at this. Strangely, the last commit removed the rapids libraries from the base cuda-images. We're adding them back in.

Re: Anyone using RAPIDS and cuGraph on a current runtime?

Prabakar — Wed, 17 Nov 2021 17:31:53 GMT

Thanks for the update @Brian Gue . Hope, adding the rapids libraries to the cuda-image helps to resolve your issue.