Hello, I am trying to run jobs with a JAR task type using databricks on AWS on an all-purpose cluster. The issue I'm facing is that the job will complete the first run successfully but on any subsequent runs, it will fail. I have to restart my cluster to get it to run properly and then it will fail on the second run again.
org.apache.spark.sql.AnalysisException: Undefined function: <function_name>. This function is neither a built-in/temporary function, nor a persistent function that is qualified as <function_name>.
at org.apache.spark.sql.errors.QueryCompilationErrors$.noSuchFunctionError(QueryCompilationErrors.scala:2025)
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$25.applyOrElse(Analyzer.scala:2919)
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$25.applyOrElse(Analyzer.scala:2905)
The exception is occurring at a point where I use a UDF registered in spark. The UDF is registered in spark at the start of each job so I'm not sure why this issue is occurring in subsequent job runs on the same cluster.
