CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND randomly appears
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a month ago
[CHECKPOINT_RDD_BLOCK_ID_NOT_FOUND] Checkpoint block not found! Either the executor that originally checkpointed this partition is no longer alive, or the original RDD is unpersisted.
After switching from reliable checkpoint() to localCheckpoint() to improve performance, my job started failing intermittently with the following error on Spark 3.5.2. The failures are not consistent sometimes the job completes successfully, and other times it fails at the same stage, which makes it hard to reproduce and debug.
- Would provisioning a larger cluster (more executor memory/disk) keep the local checkpoint data alive longer and avoid these failures?
- Labels:
-
Delta Lake
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a month ago
Its a issue due to the fundamental difference in the handling of data durability by checkpoint and localCheckpoint. Provisioning a larger cluster will not reliably solve this problem as the issue is about executor lifecycle & not capacity.
localCheckpoint data is generally tied to the executor's existence & not its size. Checkpoint data is gone regardless of how much memory or disk it had when the executor disappears. Larger cluster might reduce failures if executors were dropped due to Out-of-memory errors & Disk spill pressure. It won't prevent failures caused due to dynamic allocation, autoscaling or task failures.
You can use the reliable checkpoint for production workloads with typical Databricks cluster configurations (autoscaling etc) or use persist in MEMORY_AND_DISK mode. If you require local check point consider disabling autoscaling with large executor timeout and ensure downstream stages consume checkpointed data quickly.
