@brianjorge178 @cartergray70543 Thanks for your thoughts. I had a long chat with Databricks support team, and here is where have landed so far if it interests you:

What we believed. The pipelines committed every few minutes, so each table's transaction log folder grew to 15,000 to 22,000 files. We believed every read had to scan that whole folder, 1,000 files per LIST call, and that this is where the cost happened.

What is still true. The spike was real and LIST calls were the cost: 95% of the request volume. It started when the continuous pipelines went live and stopped the day they were stopped. The log folders did nloat, and shorter log retention is still worth doing. Our fix (triggered schedules) is still the right fix either way.

What we learned from today's update. Databricks checked the Spark logs on our busiest table. Reads were never scanning the whole folder; they jump to the last checkpoint and read only the few files after it.

So their theory diverges from ours: the cost came from the CDC connector asking S3 "any new files for me?" over and over, per table, nonstop. That polling code is inside their managed connector, not something we configure, so they've escalated to their product team to decide if it's a bug or just how it works.