topic Re: Getting spark/scala versioning issues while running the spark jobs through Jar in Data Engineering

Getting spark/scala versioning issues while running the spark jobs through Jar

Monika8991 — Tue, 13 Sep 2022 07:18:06 GMT

We tried moving our scala script from standalone cluster to databricks platform.

Our script is compatible with following version:

Spark: 2.4.8 Scala: 2.11.12

The databricks cluster has spark/scala following with version:

Spark: 3.2.1. Scala: 2.12

1: we are not able to run the script through Jar in "Yarn" mode

2: Getting issues while parsing dates

Few date functions were not working on the current version of spark so we had to change the timeParserPolicy in configuration when passed.

Corresponding Error: Fail to parse '2022-07-07 02:14:26.233' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.

3: Getting issue in using ".toList" function.

var df2=df.select("clicked_at").distinct().map(f=>f.getDate(0)).collect().toList

(this toList function is causing error)

Corresponding Error: Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps

Any help on this will be really helpful

Re: Getting spark/scala versioning issues while running the spark jobs through Jar

-werners- — Tue, 13 Sep 2022 09:10:33 GMT

indeed you have to take into account the spark version. Spark2 is not supported anymore by databricks.

This link might help.

About the toList method: it works fine here. I think the issue here is that the clicked_at column is not considered to be a valid date column.

Re: Getting spark/scala versioning issues while running the spark jobs through Jar

Anonymous — Tue, 27 Sep 2022 07:29:27 GMT

Hi @Monika Samant

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!