Hi BricksGuy,

How are you doing today?, As per my understanding, It looks like your pipeline is slowing down because it's processing too many small parquet files—over 10 million—which is causing high metadata overhead and memory issues. Since Spark has to list and open each file, performance degrades as the file count grows. To fix this, you could compact small files before ingestion using the OPTIMIZE command (if using Delta) or a Spark job to merge them. Also, try increasing cloudFiles.maxFilesPerTrigger (e.g., from 1000 to 5000) to process more files per batch without overwhelming memory. Partitioning your source data by date or another key can also help reduce file scanning time. Since you're getting GC overhead limit exceeded errors, you might need to increase cluster memory or tune memory overhead settings. Enabling Photon could also speed up processing. The key fix here is to reduce the number of small files—let me know if you need help setting that up!

Regards,

Brahma