cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Getting spark/scala versioning issues while running the spark jobs through Jar

Monika8991
New Contributor II

 We tried moving our scala script from standalone cluster to databricks platform. 

Our script is compatible with following version:

Spark: 2.4.8 Scala: 2.11.12

The databricks cluster has spark/scala following with version:

Spark: 3.2.1. Scala: 2.12

1: we are not able to run the script through Jar in "Yarn" mode

2: Getting issues while parsing dates

Few date functions were not working on the current version of spark so we had to change the timeParserPolicy in configuration when passed.

Corresponding Error: Fail to parse '2022-07-07 02:14:26.233' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.

3: Getting issue in using ".toList" function.

var df2=df.select("clicked_at").distinct().map(f=>f.getDate(0)).collect().toList

(this toList function is causing error)

Corresponding Error: Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps

Any help on this will be really helpful

2 REPLIES 2

-werners-
Esteemed Contributor III

indeed you have to take into account the spark version. Spark2 is not supported anymore by databricks.

This link might help.

About the toList method: it works fine here. I think the issue here is that the clicked_at column is not considered to be a valid date column.

Anonymous
Not applicable

Hi @Monika Samant​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.