I have to optimise our initial load, for that I want to perform batch inserts while loading data.

Databricks_Work
New Contributor II

No

Lakshay
Databricks Employee
Databricks Employee

If you want to optimize your job, you should identify which part of the job is taking time. You might want to take a look at the spark UI and identify the operations that are taking time.

while writing data from spark dataframe to PostgreSQL table , it is taking time for large tables which approx, contains 10million records

Lakshay
Databricks Employee
Databricks Employee

How many tasks do you see in the Spark UI for the write operation?