Its a issue due to the fundamental difference in the handling of data durability by checkpoint and localCheckpoint. Provisioning a larger cluster will not reliably solve this problem as the issue is about executor lifecycle & not capacity.
localCheckpoint data is generally tied to the executor's existence & not its size. Checkpoint data is gone regardless of how much memory or disk it had when the executor disappears. Larger cluster might reduce failures if executors were dropped due to Out-of-memory errors & Disk spill pressure. It won't prevent failures caused due to dynamic allocation, autoscaling or task failures.
You can use the reliable checkpoint for production workloads with typical Databricks cluster configurations (autoscaling etc) or use persist in MEMORY_AND_DISK mode. If you require local check point consider disabling autoscaling with large executor timeout and ensure downstream stages consume checkpointed data quickly.
