check whether the latest Databricks runtime has an equivalent S3 listing optimization. If not, catalog-managed commits or reducing commit frequency/retention seem to be the only practical ways to control the request costs without much complexity.