What does "ScalaDriverLocal: User Code Compile error" mean?

[cozos]

22/11/30 01:45:31 WARN ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set()
 
22/11/30 01:50:14 INFO Utils: resolved command to be run: WrappedArray(getconf, PAGESIZE)
22/11/30 01:50:15 WARN ScalaDriverLocal: User Code Compile error: command--1:1: error: object beam is not a member of package org.apache
org.apache.beam.runners.spark.SparkPipelineRunner.main(Array())

I have submitted a Jar job and my error looks like that. What does it mean to get a "User Code Compile error"?

I am curious:

• What is ScalaDriverLocal: User Code Compile doing?
• What is causing ScalaDriverLocal: User Code Compile to fail?
• What does it mean for ScalaDriverLocal: User Code Compile to fail? How can the pipeline continue to run after something like that?

[-werners-]

the error is pretty clear:

user submitted code makes the spark driver fail (scaladriverlocal).

The error explains that 'beam' does not exist. You use Apache Beam Sparkrunner in your submitted code but it seems the package is not installed on the cluster (or not imported).

https://beam.apache.org/documentation/runners/spark/

[cozos]

Hi @werners (Customer)​ thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class:

 

I guess what I'm trying to understand is what does ScalaDriverLocal: User Code Compile mean?

My code is already compiled - it is a JAR containing compiled Java byte code. So how can compilation fail on an already compiled JAR?

[cozos]

Hi @Werner Stinckens​ thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class:

 

I guess what I'm trying to understand is what does ScalaDriverLocal: User Code Compile mean?

My code is already compiled - it is a JAR containing compiled Java byte code. So how can compilation fail on an already compiled JAR?

[-werners-]

Well I am looking at

ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set()

If the library did not succeed in installing, it cannot compile.

You mention beam being included in the jar, but are you sure it is a complete jar (the forum does not allow met to use the word f-a-t :-)?

Also can you check this:

https://kb.databricks.com/en_US/jobs/job-fails-no-library

[cozos]

Yup it is an Uber JAR. I guess I'm trying to understand what it means to install a library. What is Set()? java.lang.Set()? Why are Java stdlib classes being installed? Why would Java stdlib classes *fail* to be installed? What is being compiled? A Java application? How does my already compiled JAR play into this?