Noisy error in Databricks.

alvaro_databric — Fri, 15 Sep 2023 10:31:35 GMT

Hello all,

I need some help figuring out the exact cause of this error. While in the middle of a complex scala transformation notebook in Azure Databricks, from time to time i get an error which looks like this, can someone help?

Job aborted due to stage failure: $linee899685e104341378f258eaf5a3e740579/$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.class

23/09/15 10:15:32 ERROR Uncaught throwable from user code: org.apache.spark.SparkException: Job aborted due to stage failure: Task 59 in stage 141.0 failed 4 times, most recent failure: Lost task 59.3 in stage 141.0 (TID 14396) (10.8.24.23 executor 4): org.apache.spark.repl.RemoteClassLoaderError: $linee899685e104341378f258eaf5a3e740579/$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.class

at org.apache.spark.repl.ExecutorClassLoader$$anon$1.toClassNotFound(ExecutorClassLoader.scala:156)

at org.apache.spark.repl.ExecutorClassLoader$$anon$1.read(ExecutorClassLoader.scala:143)

at java.io.FilterInputStream.read(FilterInputStream.java:107)

at org.apache.spark.repl.ExecutorClassLoader.readAndTransformClass(ExecutorClassLoader.scala:263)

at org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:217)

at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:115)

at java.lang.ClassLoader.loadClass(ClassLoader.java:419)

at java.lang.ClassLoader.loadClass(ClassLoader.java:352)

at java.lang.Class.getDeclaredFields0(Native Method)

at java.lang.Class.privateGetDeclaredFields(Class.java:2583)

at java.lang.Class.getDeclaredField(Class.java:2068)

at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1871)

at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79)

at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506)

at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494)

at java.security.AccessController.doPrivileged(Native Method)

... 209 more

Re: Noisy error in Databricks.

alvaro_databric — Mon, 18 Sep 2023 08:02:35 GMT

Hi,

Cluster configuration, user code and input data has remain unchanged. I am indeed running concurrent jobs using the same source table and generate an output table for every individual job. Can you elaborate on what you mean with resource contention? I do not understand how this applies to Databricks jobs.

Thanks in advance!

topic Re: Noisy error in Databricks. in Data Engineering

Noisy error in Databricks.

Re: Noisy error in Databricks.