topic Re: Delta Log checkpoints not being created? in Data Engineering

Delta Log checkpoints not being created?

442027 — Thu, 06 Jul 2023 01:41:07 GMT

It is mentioned in the delta protocol that checkpoints for delta tables are created every 10 commits - however when I modify a table after >10 separate operations (producing >10 separate json files in the _delta_log directory), no checkpoint files are created. Are there specific conditions under which checkpoint files are created (and not just every 10 commits); i.e. certain operations, data size, etc.?

My concern is that if checkpoints aren't created, then delta logs aren't cleaned up, and if that happens does that mean the metadata for my tables will grow infinitely large over time?

The delta tables I created were done by executing (where the storage location is s3):

```

df.write.format("delta").saveAsTable(name="<table>", path="<s3_path>", mode="overwrite", overwriteSchema=True)

```

Re: Delta Log checkpoints not being created?

Vinay_M_R — Thu, 06 Jul 2023 09:21:17 GMT

As the latest update now checkpointing of delta tables are created for every 100 commits. This is done for some improvement purpose.

If you want to have a checkpoint file for delta table for every 10 commits or after any desired commits. You can customize it using below configuration:

"delta.checkpointInterval"

Syntax:

alter table <table_name> SET TBLPROPERTIES ("delta.checkpointInterval" = "10")

you can set the desired checkpointing interval and you need to alter the table with this and set the checkpoint interval under set table properties shown as above "delta.checkpointInterval" = "10"

Re: Delta Log checkpoints not being created?

442027 — Thu, 06 Jul 2023 19:04:57 GMT

Tested and confirmed that it's every 100 commits by default. Thanks that makes a lot of sense!