Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in Delta

[elgeo]

Could someone provide any insight?

[User16752240150]

Every 10 transactions json files in the _delta_log are converted to parquet files. The .crc file is a checksum added to prevent corruption if a parquet file is corrupted in flight

[Dineshvishe]

crc is used for checksum data and data verification

[saurabh18cs]

CRC (Cyclic Redundancy Check) files in Delta Lake logs play a crucial role in ensuring data integrity and consistency. They are used to verify the integrity of the data files and transaction logs. Here's how CRC files help with transaction control in Delta Lake:

 

Role of CRC Files in Delta Lake

1. Data Integrity: CRC files store checksums for data files and transaction logs. These checksums are used to verify that the files have not been corrupted or tampered with.
2. Transaction Consistency: Delta Lake uses CRC files to ensure that all parts of a transaction are consistent. If any part of a transaction is corrupted, the CRC check will fail, and the transaction will be rolled back or flagged for correction.
3. Error Detection: CRC files help detect errors in data files and transaction logs. When a file is read, its checksum is compared with the stored CRC value. If they do not match, it indicates that the file has been corrupted.
4. Efficient Validation: CRC checksums provide a lightweight and efficient way to validate the integrity of files without having to read the entire file content. This speeds up the validation process, especially for large datasets.

[Panda]

@brickster_2018  @elgeo 

CRC ensures: Correctness, Recovery, Consistency

Checksum Verification
Read Validation
Consistent Transactions
Recovery Mechanism
Commit Optimization

It(CRC) ensures data isn't corrupted during storage or transfer, verifies consistency during reads, maintains atomicity by checking file integrity, triggers recovery on mismatch & speeds up validation without full scans.

[VasuBajaj]

A .CRC file (Cyclic Redundancy Check) is an internal checksum file used by Spark (and Hadoop) to ensure data integrity when reading and writing files.

• Data Integrity Check – .CRC files store checksums of actual data files. When reading a file, Spark/Hadoop verifies the checksum to detect corruption.
• Auto-Generated by HDFS – When a file is written to an HDFS-backed storage (like DBFS in Databricks or S3 with Hadoop connectors), Hadoop’s FileSystem API may generate .CRC files.
• Prevents Partial/Corrupt Reads – If a .CRC file exists, Spark ensures the data file hasn’t been tampered with.

  If you're using Databricks with S3 or ADLS and don't want .CRC files, you can disable checksum verification:

  spark.conf.set("spark.hadoop.fs.s3a.fast.upload", "true") spark.conf.set("spark.hadoop.fs.s3a.fast.upload.buffer", "disk")