cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

What does "ScalaDriverLocal: User Code Compile error" mean?

cozos
New Contributor III
22/11/30 01:45:31 WARN ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set()
 
22/11/30 01:50:14 INFO Utils: resolved command to be run: WrappedArray(getconf, PAGESIZE)
22/11/30 01:50:15 WARN ScalaDriverLocal: User Code Compile error: command--1:1: error: object beam is not a member of package org.apache
org.apache.beam.runners.spark.SparkPipelineRunner.main(Array())

I have submitted a Jar job and my error looks like that. What does it mean to get a "User Code Compile error"?

I am curious:

  • What is ScalaDriverLocal: User Code Compile doing?
  • What is causing ScalaDriverLocal: User Code Compile to fail?
  • What does it mean for ScalaDriverLocal: User Code Compile to fail? How can the pipeline continue to run after something like that?

6 REPLIES 6

-werners-
Esteemed Contributor III

the error is pretty clear:

user submitted code makes the spark driver fail (scaladriverlocal).

The error explains that 'beam' does not exist. You use Apache Beam Sparkrunner in your submitted code but it seems the package is not installed on the cluster (or not imported).

https://beam.apache.org/documentation/runners/spark/

cozos
New Contributor III

Hi @werners (Customer)​ thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class:

Screen 

I guess what I'm trying to understand is what does ScalaDriverLocal: User Code Compile mean?

My code is already compiled - it is a JAR containing compiled Java byte code. So how can compilation fail on an already compiled JAR?

Kaniz
Community Manager
Community Manager

Hi @Arwin Tio​, We haven’t heard from you since the last response from @Werner Stinckens​, and I was checking back to see if their suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

cozos
New Contributor III

Hi @Werner Stinckens​ thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class:

Screen Shot 2022-12-01 at 11.45.54 AM 

I guess what I'm trying to understand is what does ScalaDriverLocal: User Code Compile mean?

My code is already compiled - it is a JAR containing compiled Java byte code. So how can compilation fail on an already compiled JAR?

-werners-
Esteemed Contributor III

Well I am looking at

ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set()

If the library did not succeed in installing, it cannot compile.

You mention beam being included in the jar, but are you sure it is a complete jar (the forum does not allow met to use the word f-a-t :-)?

Also can you check this:

https://kb.databricks.com/en_US/jobs/job-fails-no-library

cozos
New Contributor III

Yup it is an Uber JAR. I guess I'm trying to understand what it means to install a library. What is Set()? java.lang.Set()? Why are Java stdlib classes being installed? Why would Java stdlib classes *fail* to be installed? What is being compiled? A Java application? How does my already compiled JAR play into this?