Hi DBStudent,

How are you doing today?, As per my understanding, You're absolutely right to be thinking about this now—your current load time isn't bad for 3 GB, but the real problem is the huge number of small files (110k!), not the data size itself. When you scale to 20–50 TB, this same approach will become a bottleneck. Spark slows down a lot when it has to scan and manage tons of tiny files, because it spends more time on file metadata than processing data. A good fix is to compact the files first by reading them in and writing them out with fewer, larger files using coalesce(). You could also look into using Autoloader, which is built for large file counts and can help with future scaling. Also, think about adjusting your cluster setup or parallelizing the work by table across workflows when you go bigger. Let me know if you’d like a sample compaction or migration setup—I’d be happy to help!

Regards,

Brahma