topic Re: What does "ScalaDriverLocal: User Code Compile error" mean? in Data Engineering

What does "ScalaDriverLocal: User Code Compile error" mean?

cozos — Thu, 01 Dec 2022 05:06:46 GMT

22/11/30 01:45:31 WARN ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set()
 
22/11/30 01:50:14 INFO Utils: resolved command to be run: WrappedArray(getconf, PAGESIZE)
22/11/30 01:50:15 WARN ScalaDriverLocal: User Code Compile error: command--1:1: error: object beam is not a member of package org.apache
org.apache.beam.runners.spark.SparkPipelineRunner.main(Array())

I have submitted a Jar job and my error looks like that. What does it mean to get a "User Code Compile error"?

I am curious:

What is ScalaDriverLocal: User Code Compile doing?
What is causing ScalaDriverLocal: User Code Compile to fail?
What does it mean for ScalaDriverLocal: User Code Compile to fail? How can the pipeline continue to run after something like that?

Re: What does "ScalaDriverLocal: User Code Compile error" mean?

-werners- — Thu, 01 Dec 2022 10:25:23 GMT

the error is pretty clear:

user submitted code makes the spark driver fail (scaladriverlocal).

The error explains that 'beam' does not exist. You use Apache Beam Sparkrunner in your submitted code but it seems the package is not installed on the cluster (or not imported).

https://beam.apache.org/documentation/runners/spark/

Re: What does "ScalaDriverLocal: User Code Compile error" mean?

cozos — Thu, 01 Dec 2022 21:35:53 GMT

Hi @Werner Stinckens thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class:

I guess what I'm trying to understand is what does ScalaDriverLocal: User Code Compile mean?

My code is already compiled - it is a JAR containing compiled Java byte code. So how can compilation fail on an already compiled JAR?

Re: What does "ScalaDriverLocal: User Code Compile error" mean?

cozos — Thu, 01 Dec 2022 21:37:06 GMT

Hi @werners (Customer) thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class:

I guess what I'm trying to understand is what does ScalaDriverLocal: User Code Compile mean?

My code is already compiled - it is a JAR containing compiled Java byte code. So how can compilation fail on an already compiled JAR?

Re: What does "ScalaDriverLocal: User Code Compile error" mean?

-werners- — Fri, 02 Dec 2022 09:21:11 GMT

Well I am looking at

ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set()

If the library did not succeed in installing, it cannot compile.

You mention beam being included in the jar, but are you sure it is a complete jar (the forum does not allow met to use the word f-a-t :-)?

Also can you check this:

https://kb.databricks.com/en_US/jobs/job-fails-no-library

Re: What does "ScalaDriverLocal: User Code Compile error" mean?

cozos — Sat, 03 Dec 2022 05:39:16 GMT

Yup it is an Uber JAR. I guess I'm trying to understand what it means to install a library. What is Set()? java.lang.Set()? Why are Java stdlib classes being installed? Why would Java stdlib classes *fail* to be installed? What is being compiled? A Java application? How does my already compiled JAR play into this?