shan_chandra
Databricks Employee
Databricks Employee

@AH  - we can try out the config 

if read or fetch from postgres is slow , we can increase the fetchsize , numPartitions (to increase parallelism). kindly try to do a df.count() to check on slowness. 

https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html

 

 If the write is slow, kindly try to a write the data to a temp table first before merge to see if this is an issue due to merge. 

View solution in original post