Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2024 12:51 PM
@AH - we can try out the config
if read or fetch from postgres is slow , we can increase the fetchsize , numPartitions (to increase parallelism). kindly try to do a df.count() to check on slowness.
https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html
If the write is slow, kindly try to a write the data to a temp table first before merge to see if this is an issue due to merge.