Inquiry on GraphFrame Library Upgrade Timeline for Databricks Runtime for Machine Learning

[toproximahk]

Thanks for the Databricks community and maintaining such a valuable platform.

I would like to inquire if there is a planned timeline for upgrading the GraphFrame library. We’ve noticed that the latest release on GitHub is v0.9.3, while the Databricks Runtime for Machine Learning (17.3 LTS) is still using v0.8.4-db1-spark3.5.

We’re particularly interested in recent updates such as the early stopping feature in Pregel (PR #550):

feat: add early stopping to Pregel by SemyonSinchenko · Pull Request #550 · graphframes/graphframes ...

Releases · graphframes/graphframes

Databricks Runtime 17.3 LTS for Machine Learning | Databricks on AWS

Thanks!

[toproximahk]

For PySpark.

[nayan_wylde]

I don't see any dates for it. But you can try this work around.

If you need access to the latest GraphFrames features

Manual Installation: You can manually install the GraphFrames v0.9.3 JAR in your cluster.

• Download from GraphFrames GitHub Releases
• Upload to DBFS or a Unity Catalog volume
• Attach via cluster libraries or init scripts

[Sem-Sinchenko]

You can try to add to your cluster mvn dependency manually ... For example, for spark 3.5.x it will be like:

io.graphframes:graphframes-spark3_2.12:0.10.0

and add a PyPi dependency graphframes-py. Adding maven coordinates should download and install all the JVM dependencies.

But most probably it won't work on DBR ML runtimes because you will have in CP two differently named graphframes JARs, but with the same namespace and barely anyone will tell you how it will be resolved in runtime... I think the best way is just using generic runtime instead of DBR ML.