- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2025 09:00 AM
Best practices / Fixes
1. Clean up the checkpoint directory before restart
If you know the stream can safely start from scratch or reprocess data:
Delete the S3 checkpoint path before restarting.
This ensures no stale 0.zip files remain.
dbutils.fs.rm(checkpoint_path, True)
2. Use a unique checkpoint location per stream
Ensure each distinct streaming query (even if reading from the same source) has a unique checkpoint path.
This avoids conflicts from multiple jobs writing to the same RocksDB state store location.
checkpoint_path = f"s3://xxx-datalake-binary/event-types/checkpoint/{uuid.uuid4()}"
3. Enable state store cleanup
You can configure Structured Streaming to clean up old state files, reducing leftover file conflicts:
spark.conf.set("spark.sql.streaming.stateStore.rocksdb.cleanupCheckpoint", "true")
Recommended approach for production
Always use unique checkpoint locations for each streaming application
Clean up checkpoints from failed runs before restarting