I have to optimise our initial load, for that I want to perform batch inserts while loading data.

[Databricks_Work]

No

[Lakshay]

If you want to optimize your job, you should identify which part of the job is taking time. You might want to take a look at the spark UI and identify the operations that are taking time.

[Databricks_Work]

while writing data from spark dataframe to PostgreSQL table , it is taking time for large tables which approx, contains 10million records

[Lakshay]

How many tasks do you see in the Spark UI for the write operation?

[Kaniz]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!