Hi everyone, I’m working with around 22,000 Unity Catalog external Delta tables, and my requirement is to execute DESCRIBE HISTORY table_name LIMIT 1 for each table and append the latest record into a single consolidated table. I’ve already tried multiple optimization approaches such as multithreading using ThreadPoolExecutor, batching tables, and tuning parameters like max workers and batch size. With these optimizations, I was able to process around 2,200 tables in about 4 minutes, but when I scaled to 12,000 tables, it took nearly 50 minutes, and performance does not scale linearly. My expectation was that since each DESCRIBE HISTORY call roughly takes around 0.8 seconds, running them in parallel with about 32 workers should ideally complete in approximately 550–700 seconds (around 10–12 minutes). However, in practice, increasing parallelism beyond a certain level does not reduce execution time significantly, and I’m hitting a bottleneck. I want to understand whether it is realistically possible to process 22k tables within ~20 minutes using this approach, or if there are better Databricks-native or Delta Lake alternatives to avoid per-table execution.  Is there any more efficient pattern for large-scale metadata extraction like this other than splitting into multiple jobs?