Failure starting repl

[liamod_1]

Hi, we have several clusters that keep giving this error:

Failure starting repl. Try detaching and re-attaching the notebook.

All the investigation I've done points to this issue being related to the number of concurrent connections but we only have 1 notebook attached to some of these clusters. It also seems like the cluster will work for a couple of hours and then not some other times.

Any assistance would be greatly appreciated.

[Debayan]

Hi, Do you see the errors in event logs? Could you also please check the driver logs for the elaborated error logs and provide it here?

Also, please tag @Debayan​ with your next comment so I will be getting notified.

[liamod_1]

Hi @Debayan Mukherjee​ In the event log I'm seeing an entry saying the Meta-store is down.

DRIVER_HEALTHY 2023-06-15 10:24:21 SAST Driver is healthy.

METASTORE_DOWN 2023-06-15 10:14:20 SAST Metastore is down.

DRIVER_HEALTHY 2023-06-15 10:08:21 SAST Driver is healthy.

RUNNING 2023-06-15 10:08:13 SAST Cluster is running.

INIT_SCRIPTS_FINISHED 2023-06-15 10:07:48 SAST Finished init scripts execution.

INIT_SCRIPTS_STARTED 2023-06-15 10:05:45 SAST Starting init scripts execution.

In root of the traceback I see:

Caused by: java.lang.Throwable: Too many connections

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:856)

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:777)

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:451)

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1103)

   ... 159 more

Thanks for reaching out!

[liamod_1]

@Debayan Mukherjee​ And at the top of the stack trace:

java.lang.Exception: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

[Debayan]

Hi @Liam O'Donoghue​ , Could you please confirm if you are in ST or E2 shard?

[liamod_1]

@Debayan Mukherjee​ I'm unsure so I have messaged our databricks contact. But we have been getting notifications about workspace migrations. Could this be related to discontinued support for ST? Thanks.

[Aviral-Bhardwaj]

most of the time this happen due to Library conflicts please check your library and try to clone that cluster then run

[Anonymous]

Hi @Liam O'Donoghue​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

[liamod_1 · Accepted solution]

@Aviral Bhardwaj​ thanks, this seemed to fix the issue, we had an innit script that was potentially conflicting with UI set libraries (in cluster settings).