topic Can use graphframes DBR 14.3 in Data Engineering

Can use graphframes DBR 14.3

Henrik_ — Wed, 14 Aug 2024 08:01:41 GMT

I get the following error when trying to run GraphFrame on DBR 14.3. Anyone has an idea of how I can solve this?

"""

import pyspark.sql.functions as F

from graphframes import GraphFrame

vertices = spark.createDataFrame([

("a", "Alice", 34),

("b", "Bob", 36),

("c", "Charlie", 30),

("d", "David", 29),

("e", "Esther", 32),

("f", "Fanny", 36),

("g", "Gabby", 60)],

["id", "name", "age"])

edges = spark.createDataFrame([

("a", "b", "friend"),

("b", "c", "follow"),

("c", "b", "follow"),

("f", "c", "follow"),

("e", "f", "follow"),

("e", "d", "friend"),

("d", "a", "friend"),

("a", "e", "friend")],

["src", "dst", "relationship"])

g = GraphFrame(vertices, edges)

display(g)

But I get this error:

[ATTRIBUTE_NOT_SUPPORTED] Attribute `sql_ctx` is not supported.

File <command-1696281758644499>, line 22 1 vertices = spark.createDataFrame([ 2 ("a", "Alice", 34), 3 ("b", "Bob", 36), (...) 8 ("g", "Gabby", 60)], 9 ["id", "name", "age"]) 11 edges = spark.createDataFrame([ 12 ("a", "b", "friend"), 13 ("b", "c", "follow"), (...) 19 ("a", "e", "friend")], 20 ["src", "dst", "relationship"]) ---> 22 g = GraphFrame(vertices, edges) 23 display(g)

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/graphframes/graphframe.py:63, in GraphFrame.__init__(self, v, e) 61 self._vertices = v 62 self._edges = e ---> 63 self._sqlContext = v.sql_ctx 64 self._sc = self._sqlContext._sc 65 self._jvm_gf_api = _java_api(self._sc)

File /databricks/spark/python/pyspark/sql/connect/dataframe.py:1757, in DataFrame.__getattr__(self, name) 1754 # END-EDGE 1756 if name not in self.columns: -> 1757 raise PySparkAttributeError( 1758 error_class="ATTRIBUTE_NOT_SUPPORTED", message_parameters={"attr_name": name} 1759 ) 1761 return self._col(name)

"""

Re: Can use graphframes DBR 14.3

jctanner — Thu, 22 Aug 2024 21:42:29 GMT

I am having the same issue. Did you ever determine how to solve this?

Re: Can use graphframes DBR 14.3

-werners- — Fri, 23 Aug 2024 06:56:06 GMT

What type of cluster do you use, and what version of graphframes?

Re: Can use graphframes DBR 14.3

Henrik_ — Fri, 23 Aug 2024 07:14:16 GMT

I'm running on 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12), 1 driver (n2-highmem 😎 64 GB Memory, 8 Cores, 1-8 Workers (n2-highmem-4) 32-256 GB Memory 4-32 Cores. My version of graphframes is 0.6.

Re: Can use graphframes DBR 14.3

-werners- — Fri, 23 Aug 2024 07:32:35 GMT

Please install the latest graphframes version.

0.6 has only spark 2 support.
Also in the latest version the deprecated sql context is removed:
https://github.com/graphframes/graphframes/releases

Re: Can use graphframes DBR 14.3

jctanner — Fri, 23 Aug 2024 16:08:28 GMT

Thanks for the response -werners-. Version 0.8.3 installed via https://pypi.org/project/graphframes-latest/ gives a different error: AttributeError: 'SparkSession' object has no attribute '_sc'. No version above 0.6 is available via %pip install graphframes --upgrade.

Note that I am using Serverless compute.

Re: Can use graphframes DBR 14.3

-werners- — Mon, 26 Aug 2024 06:18:24 GMT

Serverless compute might be the issue here.
Can you deploy a classic compute cluster and install graphframes?

Re: Can use graphframes DBR 14.3

jctanner — Tue, 03 Sep 2024 22:29:38 GMT

Hi, sorry for the delay, yes this was done several weeks ago but my issue is with deploying it on Serverless. Also we are unable to deploy custom clusters currently due to a separate issue.

Re: Can use graphframes DBR 14.3

-werners- — Wed, 04 Sep 2024 06:26:59 GMT

Serverless compute has limitations, like installing libraries. So at the moment that won't be possible.
Oldschool clusters have way more configuration possibilities, so hopefully you can fix the issue you experience on deploying clusters.

Re: Can use graphframes DBR 14.3

Snag — Fri, 31 Jan 2025 19:01:17 GMT

Hi Guys, even with the classic compute with version 14.3LTS, I'm getting the _sc error mentioned above, pls let me know if you're able to fix the issue

Re: Can use graphframes DBR 14.3

tsekityam_2 — Mon, 14 Jul 2025 09:19:46 GMT

graphframes-latest is not an official release of graphframes. So I would not recommend anyone to use it.

Anyway, graphframes is available in ML runtime, may be using ML runtime instead of the standard one is the best solution.

Re: Can use graphframes DBR 14.3

Miguel_CP — Sun, 20 Jul 2025 15:28:37 GMT

Through pip install gaphframes the version is 0.6 which is only supported by 2.x versions of Spark. For the latest versions of graphframes use: pip install graphframes-py. This, as of today, gets the version 0.9.2 which is fully compatible with Spark versions 3.5+

You can see the differences here between these two pip install commands:

pip install graphframes

pip install graphframes-py