The issue is because Databricks 15.4 LTS includes ws_3_5--core--core-hive-2.3__hadoop-3.2_2.12_deploy.jar library which is not compatible with Spark 3.5.x version. Spark 3.5.x version contains single argument SparkListenerApplicationEnd constructor.

Databrick 15.4 LTS includes Spark 3.5.x and ws_3_5--core--core-hive-2.3__hadoop-3.2_2.12_deploy.jar library should be compatible with Spark 3.5.x. But it is not.