Hi @theunwoke
you are relying on 16 partitions only to copy billion data by using repartition(num_cores*2) with r5d.2xlarge. Assuming with billion rows your data size is 500 GB then by default spark will create 4000 approx partitions (128MB is the default per partition split) but with repartition you have reduced it to 16 partitions only with shuffle where default is 200.
1) Use compute-optimized or storage-optimized cluster e.g.
c5d.4xlarge × 4 workers
2) Add .format("delta") to your write. e.g. .write.format("delta")
3) repartition(2000) with above cluster type
4) Modern alternative to COPY INTO (legacy) is CTAS which is usually faster for bulk loads. (My personal recommendation)
