S3 LIST costs on high commit rate Delta tables: is there a start-after option on Databricks runtime?

[binlogreader]

Hi All,

If you run streaming or continuous pipelines that commit to Delta tables on S3 every few minutes, your _delta_log folders are probably much bigger than your tables. A CDC or MERGE flow commits every cycle whether or not data arrived, and with the default 30 day log retention that adds up fast. A table committing every two minutes carries roughly 20,000 log files. We learned this when our S3 bill doubled in a month with storage flat, nearly all of it Tier-1 request charges against _delta_log prefixes.

The expensive part is the read path. S3 LIST returns 1,000 keys per call, so a version lookup against a 16,000 file log folder takes about 16 calls, and every streaming cycle does those lookups. Commit rate drives both how often you look up and how many pages each lookup costs, so request spend grows with the square of the commit rate. Speed a pipeline up 5x and this line of the bill goes up about 25x.

Commit files are named by version, the reader knows the last checkpoint, and S3 ListObjectsV2 has a start-after parameter, so one page should be enough. Open source Delta fixed exactly this in 2022 (issue #1191, shipped in Delta 2.3.0 as delta.enableFastS3AListFrom). But that flag is implemented inside the S3A filesystem, and the Databricks runtime uses its own S3 client, so as far as I can tell it does nothing there.

The platform answer seems to be catalog-managed commits, which skip the listing entirely, but streaming tables need Public Preview enrollment and materialized views are not supported yet.

So, for those running high commit rate tables on Databricks and S3:

1. Have you found any runtime or table configuration that makes _delta_log listing start at the checkpoint instead of paging the whole folder?
2. Has anyone been through the catalog-managed commits preview with streaming tables, and did it help with request costs?
3. Or is the current answer just triggered schedules, slower cadence, and a shorter delta.logRetentionDuration?

For us the workarounds cut the cost fine, but they trade freshness for money, and that tradeoff should not have to exist when the object store supports the precise lookup. If there is a knob I have missed I would like to know about it.

[cartergray70543]

check whether the latest Databricks runtime has an equivalent S3 listing optimization. If not, catalog-managed commits or reducing commit frequency/retention seem to be the only practical ways to control the request costs without much complexity.

[brianjorge178]

This is a really interesting breakdown of the S3 request-cost problem. I’d also be cautious about simply reducing the commit frequency, since that can solve the cost issue while creating a freshness problem.

I’d check the exact Databricks Runtime version and whether the newer catalog-managed commit options are available for your workload before relying on retention changes. If there’s no supported way to make the runtime start listing from the checkpoint, reducing unnecessary commits seems like the most practical workaround for now.

I’d also be interested to hear whether anyone has measured the request-cost difference after enabling catalog-managed commits in production. monkeymart.com.br isn’t related to the technical issue, but I’d keep the discussion focused on the Delta/S3 configuration and runtime behavior.

[binlogreader]

@brianjorge178 @cartergray70543 Thanks for your thoughts. I had a long chat with Databricks support team, and here is where have landed so far if it interests you:

What we believed. The pipelines committed every few minutes, so each table's transaction log folder grew to 15,000 to 22,000 files. We believed every read had to scan that whole folder, 1,000 files per LIST call, and that this is where the cost happened.

What is still true. The spike was real and LIST calls were the cost: 95% of the request volume. It started when the continuous pipelines went live and stopped the day they were stopped. The log folders did nloat, and shorter log retention is still worth doing. Our fix (triggered schedules) is still the right fix either way.

What we learned from today's update. Databricks checked the Spark logs on our busiest table. Reads were never scanning the whole folder; they jump to the last checkpoint and read only the few files after it.

So their theory diverges from ours: the cost came from the CDC connector asking S3 "any new files for me?" over and over, per table, nonstop. That polling code is inside their managed connector, not something we configure, so they've escalated to their product team to decide if it's a bug or just how it works.