Batch Doesn't Exist Failure

JordanYaker — Sat, 20 May 2023 12:54:17 GMT

I have a job that's been working perfectly fine since I deployed it earlier this month. Last night, however, one of the tasks within the job started failing with the following error:

java.lang.IllegalStateException: batch 4 doesn't exist
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$validateOffsetLogAndGetPrevOffset$1(MicroBatchExecution.scala:426)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.validateOffsetLogAndGetPrevOffset(MicroBatchExecution.scala:419)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.populateStartOffsets(MicroBatchExecution.scala:466)
	at org.apache.spark.sql.execution.streaming.MultiBatchRollbackSupport.populateStartOffsetsWithRollbackHandling(MultiBatchRollbackSupport.scala:112)
	at org.apache.spark.sql.execution.streaming.MultiBatchRollbackSupport.populateStartOffsetsWithRollbackHandling$(MultiBatchRollbackSupport.scala:79)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.populateStartOffsetsWithRollbackHandling(MicroBatchExecution.scala:57)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStreamWithListener$2(MicroBatchExecution.scala:339)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:336)
	at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:334)
	at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:77)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStreamWithListener$1(MicroBatchExecution.scala:329)
	at org.apache.spark.sql.execution.streaming.SingleBatchExecutor.execute(TriggerExecutor.scala:39)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStreamWithListener(MicroBatchExecution.scala:319)
	at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:307)
	at org.apache.spark.sql.execution.streaming.StreamExecution.$anonfun$runStream$1(StreamExecution.scala:368)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:985)
	at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:332)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$2(StreamExecution.scala:257)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.unity.EmptyHandle$.runWithAndClose(UCSHandle.scala:125)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:257)

I can't find anything in the documentation or the community posts so far. Anyone have an idea?

Re: Batch Doesn't Exist Failure

JordanYaker — Sat, 20 May 2023 18:44:01 GMT

I tried FSCK REPAIR just on the chance that it would work and it had no effect.

topic Batch Doesn't Exist Failure in Data Engineering

Batch Doesn't Exist Failure

Re: Batch Doesn't Exist Failure