Databricks Community

Saikumar_Manne · 02-17-2026

Hi everyone,We have a Databricks (Unity Catalog) pipeline where we process large datasets in Spark and need to load incremental data into a PostgreSQL target table.Our scenario is:Initial full load (~300 million rows) to PostgreSQL using bulk COPY is...

Saikumar_Manne

Hi @aleksandra_ch , thanks for the response — this is very helpful.Just to clarify our setup: we generate ~190M incremental rows in Databricks and bulk load them into a staging/temp table in PostgreSQL (via COPY from Spark). That part is relatively f...

Databricks Community

User Stats

User Activity

How to use multi-threading and batch inserts for large UPSERT to PostgreSQL from Databricks?

Re: How to use multi-threading and batch inserts for large UPSERT to PostgreSQL from Databricks?