How can I avoid this 'java.sql.SQLException: Too many connections' error?

Robbie — Sat, 15 Jan 2022 01:45:41 GMT

I'm having difficulty with a job (parent) that triggers multiple parallel runs of another job (child) in batches (e.g. 10 parallel runs per batch).

Occasionally some of the parallel "child" jobs will crash a few minutes in-- either during or immediately after cluster initialization. The crashed runs terminate with a 'Cancelled' result status.

Seemingly relevant excerpt from the log4j output:

Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:175)
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1107)
	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:502)
	at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:155)
	at org.mariadb.jdbc.Driver.connect(Driver.java:86)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:208)
	at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
	at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
	... 116 more
Caused by: java.sql.SQLException: Too many connections
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:856)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:777)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:451)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1103)
	... 123 more
22/01/14 21:24:42 WARN PythonDriverWrapper: setupRepl:ReplId-409cf-88936-53fe7-8: at the end, the status is Error(ReplId-409cf-88936-53fe7-8,org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient)
22/01/14 21:24:42 INFO DriverCorral$: Cleaning the wrapper ReplId-409cf-88936-53fe7-8 (currently in status Stopped(ReplId-409cf-88936-53fe7-8))
22/01/14 21:24:42 INFO DriverCorral$: sending shutdown signal for REPL ReplId-409cf-88936-53fe7-8
22/01/14 21:24:42 WARN PythonDriverWrapper: Repl ReplId-409cf-88936-53fe7-8 is already shutting down: Stopped(ReplId-409cf-88936-53fe7-8)
22/01/14 21:24:42 INFO DriverCorral$: sending the interrupt signal for REPL ReplId-409cf-88936-53fe7-8
22/01/14 21:24:42 INFO DriverCorral$: waiting for localThread to stop for REPL ReplId-409cf-88936-53fe7-8
22/01/14 21:24:42 INFO DriverCorral$: ReplId-409cf-88936-53fe7-8 successfully discarded

Full log4j-active output attached.

Re: How can I avoid this 'java.sql.SQLException: Too many connections' error?

Hubert-Dudek — Sat, 15 Jan 2022 15:40:28 GMT

It is MariaDB JDBC error so probably database which you are trying to connect can not handle this amount of concurrent connections (alternatively if you are not connecting to MariaDB databse, MariaDB is used also for hive metastore in your case mariadb://livongo-db.chkweekm4xjq.us-east-1.rds.amazonaws.com:3306 so if is that the case (but I don't think so) please report it to AWS)

topic Re: How can I avoid this 'java.sql.SQLException: Too many connections' error? in Data Engineering

How can I avoid this 'java.sql.SQLException: Too many connections' error?

Re: How can I avoid this 'java.sql.SQLException: Too many connections' error?