I am encountering a recurring issue while working with Delta streaming tables in my system. The error message is as follows:
com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: [DELTA_TRUNCATED_TRANSACTION_LOG] gs://cimb-prod-lakehouse/bronze-layer/icoredb/dpb_revi_loan/_delta_log/00000000000000000000.json: Unable to reconstruct state at version 899 as the transaction log has been truncated due to manual deletion or the log retention policy (delta.logRetentionDuration=3 days) and checkpoint retention policy (delta.checkpointRetentionDuration=2 days)
Context:
- I am designing a system that uses Delta format for streaming tables.
- The affected tables have very few transactions or updates, which seems to make them prone to this error.
- Upon inspecting the _delta_log directory, I noticed that only checkpoint versions 900 and 979 exist. However, the error indicates that it is trying to read from version 899
Questions:
- Why is Databricks attempting to access version 899 when the checkpoint files available start from version 900? Could this be a bug or misconfiguration in Delta Lake's automatic cleanup process?
- Is it possible that Delta Lake's log and checkpoint retention policies are prematurely removing active checkpoints for tables with minimal updates? If so, how can I adjust these settings to prevent this issue?
- What are the recommended best practices for managing retention policies (delta.logRetentionDuration and delta.checkpointRetentionDuration) for Delta tables with infrequent updates?
Additional Information:
- Retention settings:
- delta.logRetentionDuration = "3 days"
- delta.checkpointRetentionDuration = "2 days"
And even after reconfiguring the value of "logRetentionDuration" = "checkpointRetentionDuration" = 3 days or 7 days, the above error still occurs.
