@min shi :
Based on the code snippet you provided, it looks like the Databricks job is configured to fail if the Spark application exits with code 1. The run method returns an ExitCode based on the result of the Spark application. If the application fails with an exception, the method returns ExitCode.Error, otherwise, it returns ExitCode.Success.
However, in the logs you provided, it seems that the Spark application failed with an AnalysisException, which is an exception and not an exit code. This might be the reason why the job is not failing as expected. To make sure that the job fails when the Spark application exits with code 1, you can modify the run method to handle the exit code returned by the Spark application. Here is an example:
def run(args: List[String]): IO[ExitCode] = {
logger.info("Started the application")
val conf = defaultOverrides.withFallback(defaultApplication).withFallback(defaultReference)
val app = new OurRunner(conf, sparkFactory)
val exitCodeIO = IO(app.program(args).waitFor())
exitCodeIO.flatMap { exitCode =>
if (exitCode == 0) {
IO(logger.info("Success: Successfully finished running Spark application"))
.map(_ => ExitCode.Success)
} else {
IO(logger.error(s"Failure: Spark application exited with code $exitCode"))
.map(_ => ExitCode.Error)
}
}
}
With this modification, the run method returns ExitCode.Success if the Spark application exits with code 0, and ExitCode.Error if it exits with any other code, including 1.
