Can use graphframes DBR 14.3

[Henrik_]

I get the following error when trying to run GraphFrame on DBR 14.3. Anyone has an idea of how I can solve this? 

"""

import pyspark.sql.functions as F

from graphframes import GraphFrame

vertices = spark.createDataFrame([

    ("a", "Alice", 34),

    ("b", "Bob", 36),

    ("c", "Charlie", 30),

    ("d", "David", 29),

    ("e", "Esther", 32),

    ("f", "Fanny", 36),

    ("g", "Gabby", 60)],

    ["id", "name", "age"])

edges = spark.createDataFrame([

    ("a", "b", "friend"),

    ("b", "c", "follow"),

    ("c", "b", "follow"),

    ("f", "c", "follow"),

    ("e", "f", "follow"),

    ("e", "d", "friend"),

    ("d", "a", "friend"),

    ("a", "e", "friend")],

    ["src", "dst", "relationship"])

g = GraphFrame(vertices, edges)

display(g)

But I get this error: 

[ATTRIBUTE_NOT_SUPPORTED] Attribute `sql_ctx` is not supported.

File <command-1696281758644499>, line 22 1 vertices = spark.createDataFrame([ 2 ("a", "Alice", 34), 3 ("b", "Bob", 36), (...) 8 ("g", "Gabby", 60)], 9 ["id", "name", "age"]) 11 edges = spark.createDataFrame([ 12 ("a", "b", "friend"), 13 ("b", "c", "follow"), (...) 19 ("a", "e", "friend")], 20 ["src", "dst", "relationship"]) ---> 22 g = GraphFrame(vertices, edges) 23 display(g)

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/graphframes/graphframe.py:63, in GraphFrame.__init__(self, v, e) 61 self._vertices = v 62 self._edges = e ---> 63 self._sqlContext = v.sql_ctx 64 self._sc = self._sqlContext._sc 65 self._jvm_gf_api = _java_api(self._sc)

File /databricks/spark/python/pyspark/sql/connect/dataframe.py:1757, in DataFrame.__getattr__(self, name) 1754 # END-EDGE 1756 if name not in self.columns: -> 1757 raise PySparkAttributeError( 1758 error_class="ATTRIBUTE_NOT_SUPPORTED", message_parameters={"attr_name": name} 1759 ) 1761 return self._col(name)

"""

[jctanner]

I am having the same issue. Did you ever determine how to solve this?

[-werners-]

What type of cluster do you use, and what version of graphframes?

[Henrik_]

I'm running  on 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12), 1 driver (n2-highmem 😎 64 GB Memory, 8 Cores, 1-8 Workers (n2-highmem-4) 32-256 GB Memory 4-32 Cores.  My version of graphframes is 0.6.

[-werners-]

Please install the latest graphframes version.

0.6 has only spark 2 support.
Also in the latest version the deprecated sql context is removed:
https://github.com/graphframes/graphframes/releases

[jctanner]

Thanks for the response -werners-. Version 0.8.3 installed via https://pypi.org/project/graphframes-latest/ gives a different error: AttributeError: 'SparkSession' object has no attribute '_sc'. No version above 0.6 is available via %pip install graphframes --upgrade.

Note that I am using Serverless compute.

[-werners-]

Serverless compute might be the issue here.
Can you deploy a classic compute cluster and install graphframes?

[jctanner]

Hi, sorry for the delay, yes this was done several weeks ago but my issue is with deploying it on Serverless. Also we are unable to deploy custom clusters currently due to a separate issue.

[-werners-]

Serverless compute has limitations, like installing libraries.  So at the moment that won't be possible.
Oldschool clusters have way more configuration possibilities, so hopefully you can fix the issue you experience on deploying clusters.