cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Anyone using RAPIDS and cuGraph on a current runtime?

Anonymous
Not applicable

We're in the process of migrating a large graph computation workload to nvidia RAPIDS + cuGraph for GPU acceleration.

The package isn't a part of the base runtime and it is available by conda package management only, so can't be installed via init script or package list.

We're a bit surprised it isn't part of the base; it's been on cloudera for over a year, for example.

We're going down the custom docker container route and are curious - anyone else doing the same thing? Have you built a dockerfile? If it's useful to others, we'll share our dockerfile and image in the container registry.

1 ACCEPTED SOLUTION

Accepted Solutions

Prabakar
Esteemed Contributor III
Esteemed Contributor III

Hi @Brian Gue​ Databricks container services for GPU clusters is now in public preview. You can build your docker image to use it on a GPU cluster.

https://docs.databricks.com/clusters/gpu.html#databricks-container-services-on-gpu-clusters

You can find the GPU-supported docker image in Github.

https://github.com/databricks/containers/tree/master/ubuntu/gpu

View solution in original post

4 REPLIES 4

Anonymous
Not applicable

Hi @Brian Gue​! It's nice to meet you and thank you for your question. This sounds interesting and I look forward to seeing what our members come up with. If not, we will see what we can find for you.

Prabakar
Esteemed Contributor III
Esteemed Contributor III

Hi @Brian Gue​ Databricks container services for GPU clusters is now in public preview. You can build your docker image to use it on a GPU cluster.

https://docs.databricks.com/clusters/gpu.html#databricks-container-services-on-gpu-clusters

You can find the GPU-supported docker image in Github.

https://github.com/databricks/containers/tree/master/ubuntu/gpu

Anonymous
Not applicable

Thanks @Prabakar Ammeappin​ , we're looking at this. Strangely, the last commit removed the rapids libraries from the base cuda-images. We're adding them back in.

Prabakar
Esteemed Contributor III
Esteemed Contributor III

Thanks for the update @Brian Gue​ . Hope, adding the rapids libraries to the cuda-image helps to resolve your issue.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!