I have to optimise our initial load, for that I want to perform batch inserts while loading data.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2024 08:29 AM
No
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2024 08:46 AM
If you want to optimize your job, you should identify which part of the job is taking time. You might want to take a look at the spark UI and identify the operations that are taking time.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2024 09:14 AM
while writing data from spark dataframe to PostgreSQL table , it is taking time for large tables which approx, contains 10million records
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2024 09:27 AM
How many tasks do you see in the Spark UI for the write operation?