Writing data to RDS table taking more time

Policepatil — Thu, 31 Aug 2023 05:18:23 GMT

Hi,

Cluster Configuration details:

RDS Configuration Details:

I have 30 files, each file having 540000 records

I read all files and created one dataframe.

When i write dataframe(16,200,000 records) to a table it take more time nearly more than 1 hour (sometime it will fail saying "Connection time out error")

When i read all 30 files in multithreading and write dataframes to table (30 threads, 30 dataframes, each dataframe having 540000 records) it takes nearly 30 minutes without any error.

I want understand why writing one dataframe takes more time?

topic Writing data to RDS table taking more time in Data Engineering

Writing data to RDS table taking more time