Log has failed integrity check error when altering a table property

LavaLiah_85929
New Contributor II

‎12-05-2022 01:56 PM

Below is the integrity check error we are getting when trying to set the deletedRetentionFileDuration table property to 10 days.

image.pngimage 

Observation: The table data is sitting in S3. The size of all the files in S3 is in TB. There are millions of files for this table.

What is the best way to clear out the error apart from dropping and recreating the table?

Preview file
58 KB
Preview file
58 KB
0 Kudos

UmaMahesh1
Honored Contributor III

‎12-06-2022 12:11 AM

This might be because of issues in transaction logs. Since this is an external table and delta format, create or replace table should be fixing the issue. This should fix the transaction log issue.

However if the issue still persists you can contact databricks support or set this confg to skip the issue.

spark.conf.set("spark.databricks.delta.state.corruptionIsFatal", False)

Hope this helps..

Cheers.

Uma Mahesh D
0 Kudos

Hubert-Dudek
Databricks MVP

‎12-06-2022 03:20 AM

Please backup your table, then run the repair of files

FSCK REPAIR TABLE table_name

you can also try to make dry run first

FSCK REPAIR TABLE table_name DRY RUN

if data is partitioned can be helpful to refresh metastore

MSCK REPAIR TABLE mytable


My blog: https://databrickster.medium.com/

View solution in original post