topic Workaround for GraphFrames not working on Delta Live Table? in Data Engineering

Workaround for GraphFrames not working on Delta Live Table?

amartinez — Tue, 29 Nov 2022 01:22:56 GMT

According to this page, the GraphFrames package is included in the databricks runtime since at least 11.0. However trying to run a connected components algorithm inside a delta live table notebook yields the error java.lang.ClassNotFoundException: org.graphframes.GraphFramePythonAPI

I installed with pip using a magic command, but it seems that the package is not included in the cluster itself. Is there a workaround to get graphframes working on the delta live runtime? I tried adding the maven coordinates in the cluster definition but it seems DLT does not support maven libraries.

Re: Workaround for GraphFrames not working on Delta Live Table?

-werners- — Tue, 29 Nov 2022 08:44:59 GMT

have you tried an ML cluster? I think that is the key.

Re: Workaround for GraphFrames not working on Delta Live Table?

amartinez — Tue, 29 Nov 2022 15:53:54 GMT

How would I specify that I want a ML cluster? According to the Delta Live Table documentation I should not specify a runtime version....

Re: Workaround for GraphFrames not working on Delta Live Table?

-werners- — Tue, 29 Nov 2022 16:06:56 GMT

the doc you mention is specifically for the machine learning runtime.

DLT does not use that runtime and, as you correctly asked, you cannot define a runtime for DLT. So my previous answer is not an option. Sorry about that.

Right now, the only way to install libs is by using pip.

According to this doc pip should work.

But as DLT is still pretty new, it is possible that graphframes is not yet supported.

Re: Workaround for GraphFrames not working on Delta Live Table?

Senthil1 — Wed, 30 Nov 2022 13:10:33 GMT

DLT is specifically built for Data Engineering work as of now

Re: Workaround for GraphFrames not working on Delta Live Table?

Senthil1 — Wed, 30 Nov 2022 13:12:00 GMT

use MLFlow for tracking, monitoring with ML cluster for now

Re: Workaround for GraphFrames not working on Delta Live Table?

lprevost — Fri, 28 Jun 2024 15:35:19 GMT

I'm also trying to use GraphFrames inside a DLT pipeline. I get an error that graphframes not installed in the cluster. i"m using it successfully in test notebooks using the ML version of the cluster. Is there a way to use this inside a DLT job?