Transaction log integrity issue

[analyticsnerd]

Delta transaction log for one of our tables which is being written to by a Kafka Connect IcebergSinkConnector via the Unity Catalog Iceberg REST endpoint, is currently corrupted and is failing when trying to read with the below exception

ERROR:
com.databricks.sql.transaction.tahoe.DeltaIllegalStateException: [DELTA_TXN_LOG_FAILED_INTEGRITY] The transaction log has failed integrity checks. Failed verification at version 5618 of:
FileSizeHistogram mismatch in file sizes
FileSizeHistogram mismatch in file counts
Table size (bytes) - Expected: 5110912734 Computed: 5110843864
Number of files - Expected: 14624 Computed: 14623

could you please help understand what error is this and what do we need to do to fix this?

[K_Anudeep]

Hello @analyticsnerd ,

• Delta maintains a checksum file (.crc) for each committed version, along with the commit log (.json), recording the table’s file count, total size, and a distribution of file sizes (a histogram that buckets files by size). For performance, this checksum is maintained incrementally: each commit’s checksum is derived from the previous commit’s checksum plus the changes in the new commit, rather than by re-scanning the whole table.This lets the runtime know the exact shape of the table at any version without listing every file each time
  
  
• To ensure the stored checksum remains consistent with the actual table state, Delta performs an integrity check. When a snapshot is loaded, Delta reconstructs the table state from the transaction log and compares the resulting state with the stored checksum for that version. If the two do not match, Delta raises the DELTA_TXN_LOG_FAILED_INTEGRITY error and refuses to proceed, so that no operation runs on a version whose recorded state cannot be trusted.

• In your case, you will need to scan the checksum files inside the Tx. log directory of the table and see when the first checksum failure happened, and which commit created that .
  
  

• Likely causes include:

  1. The same file path was committed twice. Since the writer is an external writer, it seems like one file was re-committed/rewritten again by the external REST client, and that has caused the mismatch.
  2. The checksum was updated incrementally as if the file were new, even though that path already existed.
  3. A malformed external commit produced incorrect Add/Remove file accounting.

Recovery steps:

1. To immediately mitigate the issue, could you please set the below on your jobs to resume ingestion  spark.databricks.delta.checksum.mismatch.fatal=false

2. Alternatively, run the affected reads on non-Photon compute. The validation that raises this error is performed on Photon’s optimised read path; classic compute reads the table without it.

If you need help scanning the tx. log to identify teh corrupt CRC and the extra 1 file which was re-added, I suggest you raise a support ticket with the Databricks team to check the issue further