Efficiently Delete/Update/Insert Large Datasets of Records in PostgreSQL from Spark DataFrame

skohade1 — Wed, 21 May 2025 14:42:41 GMT

So I am migrating my ETL Process from Pentaho to Databricks I am using pyspark.

I have posted all the details here Staging Ground: How to Insert,Update,Delete data using databricks for large records in PostgreSQL from Spark DataFrame - Stack Overflow
Can anyone please help me giving the solution to optimize the way to do the operations?

Re: Efficiently Delete/Update/Insert Large Datasets of Records in PostgreSQL from Spark DataFrame

Advika — Thu, 22 May 2025 10:46:20 GMT

Hello @skohade1!

The link you provided isn't accessible. Could you please share the relevant details directly here? This will help Community members better understand your scenario.

topic Re: Efficiently Delete/Update/Insert Large Datasets of Records in PostgreSQL from Spark DataFrame in Get Started Discussions

Efficiently Delete/Update/Insert Large Datasets of Records in PostgreSQL from Spark DataFrame

Re: Efficiently Delete/Update/Insert Large Datasets of Records in PostgreSQL from Spark DataFrame