- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-07-2025 07:13 AM
I am creating a uber jar of my application with spark 3.5.0 spark depedencies and using jar submit on the cluster for execution.
As the spark libraries from the above jar "----ws_3_5--core--core-hive-2.3__hadoop-3.2_2.12_deploy.jar" are getting loaded.
the constructor SparkListenerApplicationEnd(long time, scala.Option<Object> exitCode) is creating a conflict.
In my code i have used a single argument constructor which is correct as per spark 3.5.0 but the loaded class from the above Databricks jar is expecting a constructor of 2 arguments which is not correct as per spark version 3.5.0
Constructor in spark 3.5.0
public SparkListenerApplicationEnd(long time)Constructor in "----ws_3_5--core--core-hive-2.3__hadoop-3.2_2.12_deploy.jar"
public SparkListenerApplicationEnd(long time, scala.Option<Object> exitCode)Need you inputs on this conflict, as the class "SparkListenerApplicationEnd" packed in "----ws_3_5--core--core-hive-2.3__hadoop-3.2_2.12_deploy.jar" is not same as the spark 3.5.0.