topic Log has failed integrity check error when altering a table property in Data Engineering

Log has failed integrity check error when altering a table property

LavaLiah_85929 — Mon, 05 Dec 2022 21:56:55 GMT

Below is the integrity check error we are getting when trying to set the deletedRetentionFileDuration table property to 10 days.

Observation: The table data is sitting in S3. The size of all the files in S3 is in TB. There are millions of files for this table.

What is the best way to clear out the error apart from dropping and recreating the table?

Re: Log has failed integrity check error when altering a table property

UmaMahesh1 — Tue, 06 Dec 2022 08:11:36 GMT

This might be because of issues in transaction logs. Since this is an external table and delta format, create or replace table should be fixing the issue. This should fix the transaction log issue.

However if the issue still persists you can contact databricks support or set this confg to skip the issue.

spark.conf.set("spark.databricks.delta.state.corruptionIsFatal", False)

Hope this helps..

Cheers.

Re: Log has failed integrity check error when altering a table property

Hubert-Dudek — Tue, 06 Dec 2022 11:20:13 GMT

Please backup your table, then run the repair of files

FSCK REPAIR TABLE table_name

you can also try to make dry run first

FSCK REPAIR TABLE table_name DRY RUN

if data is partitioned can be helpful to refresh metastore

MSCK REPAIR TABLE mytable