cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Java Error for installation rasterframes

jnjns
New Contributor II

Hi all,

I have followed the steps in this notebook to install rasterframes on my databricks cluster.

Eventually I am able to import the following:

from pyrasterframes import rf_ipython
from pyrasterframes.utils import create_rf_spark_session
from pyspark.sql.functions import lit 
from pyrasterframes.rasterfunctions import *

But when I run:

spark = create_rf_spark_session()

I get the following error: "java.lang.NoClassDefFoundError: scala/Product$class".

I am using a cluster with Spark 3.2.1. I also installed Java Runtime Environment 1.8.0_341, but this made no difference.

Could someone explain what went wrong? And how to solve this error?

The full error log:

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<command-2354681519525034> in <module>
      5 
      6 # Use the provided convenience function to create a basic local SparkContext
----> 7 spark = create_rf_spark_session()
 
/databricks/python/lib/python3.8/site-packages/pyrasterframes/utils.py in create_rf_spark_session(master, **kwargs)
     97 
     98     try:
---> 99         spark.withRasterFrames()
    100         return spark
    101     except TypeError as te:
 
/databricks/python/lib/python3.8/site-packages/pyrasterframes/__init__.py in _rf_init(spark_session)
     42     """ Adds RasterFrames functionality to PySpark session."""
     43     if not hasattr(spark_session, "rasterframes"):
---> 44         spark_session.rasterframes = RFContext(spark_session)
     45         spark_session.sparkContext._rf_context = spark_session.rasterframes
     46 
 
/databricks/python/lib/python3.8/site-packages/pyrasterframes/rf_context.py in __init__(self, spark_session)
     37         self._jvm = self._gateway.jvm
     38         jsess = self._spark_session._jsparkSession
---> 39         self._jrfctx = self._jvm.org.locationtech.rasterframes.py.PyRFContext(jsess)
     40 
     41     def list_to_seq(self, py_list):
 
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1566 
   1567         answer = self._gateway_client.send_command(command)
-> 1568         return_value = get_return_value(
   1569             answer, self._gateway_client, None, self._fqn)
   1570 
 
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
    115     def deco(*a, **kw):
    116         try:
--> 117             return f(*a, **kw)
    118         except py4j.protocol.Py4JJavaError as e:
    119             converted = convert_exception(e.java_exception)
 
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)
 
Py4JJavaError: An error occurred while calling None.org.locationtech.rasterframes.py.PyRFContext.
: java.lang.NoClassDefFoundError: scala/Product$class
	at org.locationtech.rasterframes.model.TileDimensions.<init>(TileDimensions.scala:35)
	at org.locationtech.rasterframes.package$.<init>(rasterframes.scala:55)
	at org.locationtech.rasterframes.package$.<clinit>(rasterframes.scala)
	at org.locationtech.rasterframes.py.PyRFContext.<init>(PyRFContext.scala:49)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:250)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 15 more

Many thanks in advance!

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group