Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
Showing results for 
Search instead for 
Did you mean: 

Java Error for installation rasterframes

New Contributor II

Hi all,

I have followed the steps in this notebook to install rasterframes on my databricks cluster.

Eventually I am able to import the following:

from pyrasterframes import rf_ipython
from pyrasterframes.utils import create_rf_spark_session
from pyspark.sql.functions import lit 
from pyrasterframes.rasterfunctions import *

But when I run:

spark = create_rf_spark_session()

I get the following error: "java.lang.NoClassDefFoundError: scala/Product$class".

I am using a cluster with Spark 3.2.1. I also installed Java Runtime Environment 1.8.0_341, but this made no difference.

Could someone explain what went wrong? And how to solve this error?

The full error log:

Py4JJavaError                             Traceback (most recent call last)
<command-2354681519525034> in <module>
      6 # Use the provided convenience function to create a basic local SparkContext
----> 7 spark = create_rf_spark_session()
/databricks/python/lib/python3.8/site-packages/pyrasterframes/ in create_rf_spark_session(master, **kwargs)
     98     try:
---> 99         spark.withRasterFrames()
    100         return spark
    101     except TypeError as te:
/databricks/python/lib/python3.8/site-packages/pyrasterframes/ in _rf_init(spark_session)
     42     """ Adds RasterFrames functionality to PySpark session."""
     43     if not hasattr(spark_session, "rasterframes"):
---> 44         spark_session.rasterframes = RFContext(spark_session)
     45         spark_session.sparkContext._rf_context = spark_session.rasterframes
/databricks/python/lib/python3.8/site-packages/pyrasterframes/ in __init__(self, spark_session)
     37         self._jvm = self._gateway.jvm
     38         jsess = self._spark_session._jsparkSession
---> 39         self._jrfctx =
     41     def list_to_seq(self, py_list):
/databricks/spark/python/lib/ in __call__(self, *args)
   1567         answer = self._gateway_client.send_command(command)
-> 1568         return_value = get_return_value(
   1569             answer, self._gateway_client, None, self._fqn)
/databricks/spark/python/pyspark/sql/ in deco(*a, **kw)
    115     def deco(*a, **kw):
    116         try:
--> 117             return f(*a, **kw)
    118         except py4j.protocol.Py4JJavaError as e:
    119             converted = convert_exception(e.java_exception)
/databricks/spark/python/lib/ in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling
: java.lang.NoClassDefFoundError: scala/Product$class
	at org.locationtech.rasterframes.model.TileDimensions.<init>(TileDimensions.scala:35)
	at org.locationtech.rasterframes.package$.<init>(rasterframes.scala:55)
	at org.locationtech.rasterframes.package$.<clinit>(rasterframes.scala)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
	at java.lang.reflect.Constructor.newInstance(
	at py4j.reflection.MethodInvoker.invoke(
	at py4j.reflection.ReflectionEngine.invoke(
	at py4j.Gateway.invoke(
	at py4j.commands.ConstructorCommand.invokeConstructor(
	at py4j.commands.ConstructorCommand.execute(
Caused by: java.lang.ClassNotFoundException: scala.Product$class
	at java.lang.ClassLoader.loadClass(
	at com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
	at java.lang.ClassLoader.loadClass(
	... 15 more

Many thanks in advance!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!