topic Enable to use library GraphFrames in Data Engineering

Enable to use library GraphFrames

sachamourier — Mon, 28 Jul 2025 15:17:06 GMT

Hello,

I am trying to install and use the library GraphFrames but keep receiving the following error: "AttributeError: 'SparkSession' object has no attribute '_sc'".

I have tried to install the library on my all-purpose cluster (Access mode: Standard). The installation works, but the code not. I am using the library version "graphframes:graphframes:0.8.4-spark3.5-s_2.13" and my spark version is 3.5.2.
I have also tried to install the library via pip install but no success either.

Does anyone know how to make it work ? I would like to avoid having to change my cluster's access mode.

Thanks a lot,

Sacha

Re: Enable to use library GraphFrames

szymon_dybczak — Mon, 28 Jul 2025 15:40:09 GMT

Hi @sachamourier ,

Maybe try to use Databricks Runtime ML which already includes an optimized installation of GraphFrames?

How to use GraphFrames on Azure Databricks - Azure Databricks | Microsoft Learn

Re: Enable to use library GraphFrames

szymon_dybczak — Mon, 28 Jul 2025 16:19:02 GMT

Hi @sachamourier ,

But if you don't want to use different runtime then you need to change access mode. In standard access mode you don't have access to SparkContext which this library requires. Hence you're getting a an error like "'SparkSession' object has no attribute '_sc'" (where _sc refers to SparkContext).

Re: Enable to use library GraphFrames

szymon_dybczak — Mon, 28 Jul 2025 16:28:06 GMT

And yes, I can confirm that it works in dedicated access mode. I've used following code:

pip install graphframes-py from functools import reduce from pyspark.sql import functions as F from graphframes import GraphFrame nodes = [ (1, "Alice", 30), (2, "Bob", 25), (3, "Charlie", 35) ] nodes_df = spark.createDataFrame(nodes, ["id", "name", "age"]) edges = [ (1, 2, "friend"), (2, 1, "friend"), (2, 3, "friend"), (3, 2, "enemy") # eek! ] edges_df = spark.createDataFrame(edges, ["src", "dst", "relationship"]) g = GraphFrame(nodes_df, edges_df)

And as you can see it works as expected:

One thing to remember, Python distribution does not include JVM-core. So I had to install also this version of library on my cluster : graphframes:graphframes:0.8.3-spark3.5-s_2.13

Re: Enable to use library GraphFrames

sachamourier — Tue, 29 Jul 2025 07:28:06 GMT

@szymon_dybczak Thanks for the responses. I indeed changed my all-purpose cluster access mode and it worked. I figured that was a nicest option than changing the runtime.

Re: Enable to use library GraphFrames

szymon_dybczak — Tue, 29 Jul 2025 07:34:08 GMT

Cool, great that it worked for you!