CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears

yanchr — Thu, 25 Jun 2026 13:19:44 GMT

[CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND] Checkpoint block not found!
Either the executor that originally checkpointed this partition is no longer alive, or the original RDD is unpersisted.

After switching from reliable checkpoint() to localCheckpoint() to improve performance, my job started failing intermittently with the following error on Spark 3.5.2. The failures are not consistent sometimes the job completes successfully, and other times it fails at the same stage, which makes it hard to reproduce and debug.

Would provisioning a larger cluster (more executor memory/disk) keep the local checkpoint data alive longer and avoid these failures?

Re: CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears

balajij8 — Thu, 25 Jun 2026 13:33:02 GMT

Its a issue due to the fundamental difference in the handling of data durability by checkpoint and localCheckpoint. Provisioning a larger cluster will not reliably solve this problem as the issue is about executor lifecycle & not capacity.

localCheckpoint data is generally tied to the executor's existence & not its size. Checkpoint data is gone regardless of how much memory or disk it had when the executor disappears. Larger cluster might reduce failures if executors were dropped due to Out-of-memory errors & Disk spill pressure. It won't prevent failures caused due to dynamic allocation, autoscaling or task failures.

You can use the reliable checkpoint for production workloads with typical Databricks cluster configurations (autoscaling etc) or use persist in MEMORY_AND_DISK mode. If you require local check point consider disabling autoscaling with large executor timeout and ensure downstream stages consume checkpointed data quickly.

topic CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears in Data Engineering

CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears

Re: CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears