Hi @VZLA ,

Thank you for your detailed suggestions and guidance.

Here are some updates and points regarding our current setup:

• The primary workloads/jobs we use are streaming jobs running on Databricks.
• We’ve already implemented Auto Optimize and Auto Compaction. Instead of partitions, we use Clustering for our data, which we believe is a viable alternative.
• Currently, we have approximately 300 jobs triggered every minute.
• Data retention is set to around 2 days, and we regularly perform vacuuming and optimization tasks.
• The setting spark.databricks.io.cache.enabled = true is also enabled across all relevant clusters.
• Yes, we do have some large tables with significant amounts of data, and we suspect these are contributing to the frequent directory listing operations. Additionally, there are no pipelines or external watchers outside Databricks writing to GCS.

Are you thinking the problem is 300 jobs being triggered continuously?