Hi @Saikumar_Manne ,

Yes, you are right by saying that numPartitions would not help if the bottleneck is on the Postgres side. 

AUTO CDC is very scalable solution which could easily accommodate 100M+ increments. I do think it would be faster than the current approach. Some best practices to speed up AUTO CDC:

  • Set up liquid clustering on the primary keys in the target table of the AUTO CDC - this will speed up the merge operation
  • Process data incrementally with auto loader or reading the table as stream
  • Use serverless for better scalability

Best regards,

View solution in original post