Hi @mjtd,
The error "Unable to reconstruct state at version 1023 as the transaction log has been truncated due to manual deletion or the log retention policy" is telling you that Delta cannot find a continuous chain of log entries from the most recent checkpoint up to the current version. Looking at your _delta_log contents, the gap between the checkpoint at version 1015 and the compacted range starting at 1016 is the clue.
Let me walk through your questions.
1. CAN SPARK READ .COMPACTED.JSON FILES?
Yes. The .compacted.json format is defined in the Delta Lake protocol as a supplemental optimization. A file named 00000000000000001016.00000000000000001021.compacted.json aggregates the actions from commit versions 1016 through 1021 into a single file. When the reader supports log compaction, it can use this file instead of reading each individual JSON commit (1016.json, 1017.json, ..., 1021.json).
However, the compacted file only replaces reading those individual commits. It does not replace checkpoints, and it does not fill in for missing files outside its range. In your case, the reader still needs:
- The checkpoint at version 1015
- Commits (or a compacted equivalent) covering versions 1016 through 1021 (which the .compacted.json provides)
- Commit 1022.json
- Commit 1023.json
If any of the individual commit files in the range were deleted but the compacted file remains, the reader should be able to use the compacted file instead, provided the Databricks Runtime version supports log compaction reads. This is controlled by the Spark config spark.databricks.delta.deltaLog.minorCompaction.useForReads (defaults to true on recent DBR versions).
The fact that you are seeing the "truncated" error suggests either:
- The runtime you are using does not support reading .compacted.json files (older DBR versions)
- There is still a gap in the log chain that even the compacted file cannot fill
2. WHY WERE THESE FILES CREATED?
Log compaction files are created automatically by Delta Lake during write operations on supported Databricks Runtime versions. This is an optimization to speed up snapshot reconstruction by reducing the number of individual JSON files the reader has to parse. The original individual commit files should still exist alongside the compacted files, as compaction does not delete the originals.
3. COULD AZURE LIFECYCLE MANAGEMENT HAVE DELETED THEM?
This is the most likely explanation. If you have an Azure Blob Storage or ADLS Gen2 lifecycle management policy configured on the storage account or container, it may be deleting or tiering files in the _delta_log directory based on age or last-modified time. The individual JSON commit files (1016.json through 1021.json) may have been deleted by such a policy while the .compacted.json file was newer and survived.
To check, go to the Azure Portal and navigate to your storage account:
- Storage Account > Data management > Lifecycle management
- Review any rules that apply to your container, especially rules targeting blobs by last modified date or by path prefix
Any lifecycle rule that deletes or tiers blobs in the _delta_log path can corrupt your Delta tables. The _delta_log directory should be excluded from all lifecycle management rules.
You can also check Azure Monitor or Storage Analytics logs to see if delete operations occurred on specific files.
4. RECOVERING THE TABLE
Since you confirmed you were able to recover by undeleting soft-deleted files, that is the correct approach when soft delete is enabled. For future protection:
a) Exclude _delta_log from lifecycle management policies. The best approach is to add a filter that skips any path containing _delta_log:
{
"rules": [
{
"name": "your-cleanup-rule",
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": ["blockBlob"],
"prefixMatch": ["your-data-path/"],
"prefixNotMatch": ["your-data-path/_delta_log/"]
}
}
}
]
}
b) Increase log retention if needed. If you want to keep transaction log files longer, set the table property:
ALTER TABLE your_catalog.your_schema.your_table
SET TBLPROPERTIES ('delta.logRetentionDuration' = 'interval 90 days');
c) Keep soft delete enabled. This saved you in this case and provides a safety net for accidental file removal.
WHY TIME TRAVEL ALSO FAILED
You mentioned that even time travel to checkpointed versions did not work. This happens because Delta Lake always reconstructs the latest snapshot first to validate the table state, regardless of which version you request via time travel. If the log chain to the current version is broken, the reconstruction fails before it can apply any time travel logic.
SUMMARY
- .compacted.json files are a valid part of the Delta protocol and Spark can read them on supported runtimes
- They aggregate multiple commits into one file but do not replace checkpoints
- The root cause was almost certainly external deletion of individual commit JSON files (likely Azure lifecycle management)
- Recovery via soft-delete undelete was the correct fix
- Prevent recurrence by excluding _delta_log from any storage lifecycle rules
Relevant documentation:
https://docs.databricks.com/en/delta/history.html
https://github.com/delta-io/delta/blob/master/PROTOCOL.md#log-compaction-files
* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.
If this answer resolves your question, could you mark it as "Accept as Solution"? That helps other users quickly find the correct fix.
