Failure starting repl

[liamod_1]

Hi, we have several clusters that keep giving this error:

Failure starting repl. Try detaching and re-attaching the notebook.

All the investigation I've done points to this issue being related to the number of concurrent connections but we only have 1 notebook attached to some of these clusters. It also seems like the cluster will work for a couple of hours and then not some other times.

Any assistance would be greatly appreciated.

[Debayan]

Hi, Do you see the errors in event logs? Could you also please check the driver logs for the elaborated error logs and provide it here?

Also, please tag @Debayan​ with your next comment so I will be getting notified.

[liamod_1]

Hi @Debayan Mukherjee​ In the event log I'm seeing an entry saying the Meta-store is down.

DRIVER_HEALTHY 2023-06-15 10:24:21 SAST Driver is healthy.

METASTORE_DOWN 2023-06-15 10:14:20 SAST Metastore is down.

DRIVER_HEALTHY 2023-06-15 10:08:21 SAST Driver is healthy.

RUNNING 2023-06-15 10:08:13 SAST Cluster is running.

INIT_SCRIPTS_FINISHED 2023-06-15 10:07:48 SAST Finished init scripts execution.

INIT_SCRIPTS_STARTED 2023-06-15 10:05:45 SAST Starting init scripts execution.

In root of the traceback I see:

Caused by: java.lang.Throwable: Too many connections

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:856)

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:777)

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:451)

   at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1103)

   ... 159 more

Thanks for reaching out!

[liamod_1]

@Debayan Mukherjee​ And at the top of the stack trace:

java.lang.Exception: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

[Kaniz]

Hi @Liam O'Donoghue​, The log entry indicates that the Metastore, a critical component of Databricks, is down. The error message suggests the issue is related to "Too many connections" in the MariaDB JDBC driver. This can occur when more connections are being made to the Metastore than it can handle.

[Debayan]

Hi @Liam O'Donoghue​ , Could you please confirm if you are in ST or E2 shard?

[liamod_1]

@Debayan Mukherjee​ I'm unsure so I have messaged our databricks contact. But we have been getting notifications about workspace migrations. Could this be related to discontinued support for ST? Thanks.

[JosephSullivan]

Any update? By the way, I'm trying to find the quickest approach to locate reputable casinos. I then discovered this https://casinosanalyzer.com/online-casinos/superace88.com website where I can play superace88 review and observe many other online slots. Visit the resources link for additional information. Here, you can engage in real-money slot machine gaming. If you're looking for online slots as well, visit this website to play slots.

[Aviral-Bhardwaj]

most of the time this happen due to Library conflicts please check your library and try to clone that cluster then run

[Anonymous]

Hi @Liam O'Donoghue​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

[liamod_1 · Accepted solution]

@Aviral Bhardwaj​ thanks, this seemed to fix the issue, we had an innit script that was potentially conflicting with UI set libraries (in cluster settings).

[Kareemlowe46]

After some initial skepticism, Barker agreed to give Plinko https://plnkgame.com a try. The game was an instant hit with both the audience and the contestants. The concept was simple but exciting - players would drop a disc down a large pegboard, and depending on which slot it landed in, they could win a cash prize ranging from a few dollars to a few thousand dollars.