topic Re: Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in De in Data Engineering

Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in Delta

brickster_2018 — Fri, 25 Jun 2021 22:41:25 GMT

Re: Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in Delta

elgeo — Tue, 25 Oct 2022 08:12:39 GMT

Could someone provide any insight?

Re: Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in Delta

User16752240150 — Mon, 09 Jan 2023 22:37:28 GMT

Every 10 transactions json files in the _delta_log are converted to parquet files. The .crc file is a checksum added to prevent corruption if a parquet file is corrupted in flight

Re: Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in De

Dineshvishe — Sun, 20 Oct 2024 15:38:55 GMT

crc is used for checksum data and data verification

Re: Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in De

saurabh18cs — Mon, 21 Oct 2024 08:58:29 GMT

CRC (Cyclic Redundancy Check) files in Delta Lake logs play a crucial role in ensuring data integrity and consistency. They are used to verify the integrity of the data files and transaction logs. Here's how CRC files help with transaction control in Delta Lake:

Role of CRC Files in Delta Lake

Data Integrity: CRC files store checksums for data files and transaction logs. These checksums are used to verify that the files have not been corrupted or tampered with.
Transaction Consistency: Delta Lake uses CRC files to ensure that all parts of a transaction are consistent. If any part of a transaction is corrupted, the CRC check will fail, and the transaction will be rolled back or flagged for correction.
Error Detection: CRC files help detect errors in data files and transaction logs. When a file is read, its checksum is compared with the stored CRC value. If they do not match, it indicates that the file has been corrupted.
Efficient Validation: CRC checksums provide a lightweight and efficient way to validate the integrity of files without having to read the entire file content. This speeds up the validation process, especially for large datasets.

Re: Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in De

Panda — Mon, 21 Oct 2024 10:27:47 GMT

@brickster_2018 @elgeo

CRC ensures: Correctness, Recovery, Consistency

Checksum Verification
Read Validation
Consistent Transactions
Recovery Mechanism
Commit Optimization

It(CRC) ensures data isn't corrupted during storage or transfer, verifies consistency during reads, maintains atomicity by checking file integrity, triggers recovery on mismatch & speeds up validation without full scans.

Re: Why do we need CRC files in Delta logs. How does CRC file help for the transaction control in De

VasuBajaj — Mon, 17 Mar 2025 05:27:34 GMT

A .CRC file (Cyclic Redundancy Check) is an internal checksum file used by Spark (and Hadoop) to ensure data integrity when reading and writing files.

Data Integrity Check – .CRC files store checksums of actual data files. When reading a file, Spark/Hadoop verifies the checksum to detect corruption.
Auto-Generated by HDFS – When a file is written to an HDFS-backed storage (like DBFS in Databricks or S3 with Hadoop connectors), Hadoop’s FileSystem API may generate .CRC files.
Prevents Partial/Corrupt Reads – If a .CRC file exists, Spark ensures the data file hasn’t been tampered with.
If you're using Databricks with S3 or ADLS and don't want .CRC files, you can disable checksum verification:
spark.conf.set("spark.hadoop.fs.s3a.fast.upload", "true") spark.conf.set("spark.hadoop.fs.s3a.fast.upload.buffer", "disk")