Can use graphframes DBR 14.3

[Henrik_]

I get the following error when trying to run GraphFrame on DBR 14.3. Anyone has an idea of how I can solve this? 

"""

import pyspark.sql.functions as F

from graphframes import GraphFrame

vertices = spark.createDataFrame([

    ("a", "Alice", 34),

    ("b", "Bob", 36),

    ("c", "Charlie", 30),

    ("d", "David", 29),

    ("e", "Esther", 32),

    ("f", "Fanny", 36),

    ("g", "Gabby", 60)],

    ["id", "name", "age"])

edges = spark.createDataFrame([

    ("a", "b", "friend"),

    ("b", "c", "follow"),

    ("c", "b", "follow"),

    ("f", "c", "follow"),

    ("e", "f", "follow"),

    ("e", "d", "friend"),

    ("d", "a", "friend"),

    ("a", "e", "friend")],

    ["src", "dst", "relationship"])

g = GraphFrame(vertices, edges)

display(g)

But I get this error: 

[ATTRIBUTE_NOT_SUPPORTED] Attribute `sql_ctx` is not supported.

File <command-1696281758644499>, line 22 1 vertices = spark.createDataFrame([ 2 ("a", "Alice", 34), 3 ("b", "Bob", 36), (...) 8 ("g", "Gabby", 60)], 9 ["id", "name", "age"]) 11 edges = spark.createDataFrame([ 12 ("a", "b", "friend"), 13 ("b", "c", "follow"), (...) 19 ("a", "e", "friend")], 20 ["src", "dst", "relationship"]) ---> 22 g = GraphFrame(vertices, edges) 23 display(g)

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/graphframes/graphframe.py:63, in GraphFrame.__init__(self, v, e) 61 self._vertices = v 62 self._edges = e ---> 63 self._sqlContext = v.sql_ctx 64 self._sc = self._sqlContext._sc 65 self._jvm_gf_api = _java_api(self._sc)

File /databricks/spark/python/pyspark/sql/connect/dataframe.py:1757, in DataFrame.__getattr__(self, name) 1754 # END-EDGE 1756 if name not in self.columns: -> 1757 raise PySparkAttributeError( 1758 error_class="ATTRIBUTE_NOT_SUPPORTED", message_parameters={"attr_name": name} 1759 ) 1761 return self._col(name)

"""

[jctanner]

I am having the same issue. Did you ever determine how to solve this?

[-werners-]

What type of cluster do you use, and what version of graphframes?

[Henrik_]

I'm running  on 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12), 1 driver (n2-highmem 😎 64 GB Memory, 8 Cores, 1-8 Workers (n2-highmem-4) 32-256 GB Memory 4-32 Cores.  My version of graphframes is 0.6.

[-werners-]

Please install the latest graphframes version.

0.6 has only spark 2 support.
Also in the latest version the deprecated sql context is removed:
https://github.com/graphframes/graphframes/releases

[jctanner]

Thanks for the response -werners-. Version 0.8.3 installed via https://pypi.org/project/graphframes-latest/ gives a different error: AttributeError: 'SparkSession' object has no attribute '_sc'. No version above 0.6 is available via %pip install graphframes --upgrade.

Note that I am using Serverless compute.

[tsekityam_2]

graphframes-latest is not an official release of graphframes. So I would not recommend anyone to use it.

Anyway, graphframes is available in ML runtime, may be using ML runtime instead of the standard one is the best solution.

[-werners-]

Serverless compute might be the issue here.
Can you deploy a classic compute cluster and install graphframes?

[jctanner]

Hi, sorry for the delay, yes this was done several weeks ago but my issue is with deploying it on Serverless. Also we are unable to deploy custom clusters currently due to a separate issue.

[-werners-]

Serverless compute has limitations, like installing libraries.  So at the moment that won't be possible.
Oldschool clusters have way more configuration possibilities, so hopefully you can fix the issue you experience on deploying clusters.

[Snag]

Hi Guys, even with the classic compute with version 14.3LTS, I'm getting the _sc error mentioned above, pls let me know if you're able to fix the issue

[Miguel_CP]

Through pip install gaphframes the version is 0.6 which is only supported by 2.x versions of Spark. For the latest versions of graphframes use: pip install graphframes-py. This, as of today, gets the version 0.9.2 which is fully compatible with Spark versions 3.5+

You can see the differences here between these two pip install commands:

pip install graphframes

pip install graphframes-py