OOM while loading a lot of data through JDBC
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2023 01:24 AM - edited 10-13-2023 01:31 AM
public void bigDataTest() throws Exception {
int rowsCount = 100_000;
int colSize = 1024;
int colCount = 12;
String colValue = "'"+"x".repeat(colSize)+"'";
String query = "select explode(sequence(1, "+rowsCount+"))," +
String.join(",", Collections.nCopies(colCount, colValue));
try (
Connection conn = dataSource.getConnection()
) {
PreparedStatement ps = conn.prepareStatement(query, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
ps.setFetchSize(1);
ResultSet rs = ps.executeQuery();
int count = 0;
while(rs.next()) {
if(count++ % 100 == 0) {
LOG.info("Count = {}", count);
}
}
}
}
With -Xmx200m I can read about 50_000 rows and after that I receive "Exception in thread "pool-12-thread-50" Exception in thread "pool-12-thread-1" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space"
The memory picture is classic for OOM:
What can I see in the heapdump: