topic Re: I have to optimise our initial load, for that I want to perform batch inserts while loading dat in Data Engineering https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56965#M30697 <P>If you want to optimize your job, you should identify which part of the job is taking time. You might want to take a look at the spark UI and identify the operations that are taking time.</P> Thu, 11 Jan 2024 16:46:23 GMT Lakshay 2024-01-11T16:46:23Z I have to optimise our initial load, for that I want to perform batch inserts while loading data. https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56956#M30692 <P>No</P> Thu, 11 Jan 2024 16:29:21 GMT https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56956#M30692 Databricks_Work 2024-01-11T16:29:21Z Re: I have to optimise our initial load, for that I want to perform batch inserts while loading dat https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56965#M30697 <P>If you want to optimize your job, you should identify which part of the job is taking time. You might want to take a look at the spark UI and identify the operations that are taking time.</P> Thu, 11 Jan 2024 16:46:23 GMT https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56965#M30697 Lakshay 2024-01-11T16:46:23Z Re: I have to optimise our initial load, for that I want to perform batch inserts while loading dat https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56968#M30699 <P>while writing data from spark dataframe to PostgreSQL table , it is taking time for large tables which approx, contains 10million records</P> Thu, 11 Jan 2024 17:14:04 GMT https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56968#M30699 Databricks_Work 2024-01-11T17:14:04Z Re: I have to optimise our initial load, for that I want to perform batch inserts while loading dat https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56974#M30701 <P>How many tasks do you see in the Spark UI for the write operation?</P> Thu, 11 Jan 2024 17:27:02 GMT https://community.databricks.com/t5/data-engineering/i-have-to-optimise-our-initial-load-for-that-i-want-to-perform/m-p/56974#M30701 Lakshay 2024-01-11T17:27:02Z