Cannot find UDF on subsequent job runs on same cluster.

[Salman1]

Hello, I am trying to run jobs with a JAR task type using databricks on AWS on an all-purpose cluster. The issue I'm facing is that the job will complete the first run successfully but on any subsequent runs, it will fail. I have to restart my cluster to get it to run properly and then it will fail on the second run again.

org.apache.spark.sql.AnalysisException: Undefined function: <function_name>. This function is neither a built-in/temporary function, nor a persistent function that is qualified as <function_name>.
	at org.apache.spark.sql.errors.QueryCompilationErrors$.noSuchFunctionError(QueryCompilationErrors.scala:2025)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$25.applyOrElse(Analyzer.scala:2919)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$25.applyOrElse(Analyzer.scala:2905)

The exception is occurring at a point where I use a UDF registered in spark. The UDF is registered in spark at the start of each job so I'm not sure why this issue is occurring in subsequent job runs on the same cluster.

[Kaniz]

Hi @Salman1, first, navigate to the corresponding job in the Databricks Jobs UI. From there, go to the Runs tab, where you can see all current and completed runs, including any that have failed. If you hover over the failed task, you can see further details such as error messages. To access even more information, click on the failed task to be directed to the Task run details page. This page displays the output of the task as well as additional information.