Hi,

The code synchronizes Databricks with PostgreSQL by identifying differences and applying INSERT, UPDATE, or DELETE operations to update PostgreSQL. The steps are as follows:

1. Read the source data in Databricks using a simple spark.sql query.
2. Read the data from PostgreSQL using the JDBC driver.
3. Perform a JOIN operation to identify differences.
4. Collect the data using .collect() (I am now trying to use .toLocalIterator()).
5. Chunk the data and iterate over it, executing DML operations using psycopg2 in batch (extras.execute_batch()), pushing a list of tuples with page_size=1000.
6. …and that’s all.

Could the issue be that psycopg2 is not an API call from Databricks, so execution is handled by the driver? Or is the .collect() operation causing a bottleneck by bringing too much data to the driver at once?