topic Re: Failure starting repl in Data Engineering

Failure starting repl

liamod_1 — Wed, 14 Jun 2023 08:38:01 GMT

Hi, we have several clusters that keep giving this error:

Failure starting repl. Try detaching and re-attaching the notebook.

All the investigation I've done points to this issue being related to the number of concurrent connections but we only have 1 notebook attached to some of these clusters. It also seems like the cluster will work for a couple of hours and then not some other times.

Any assistance would be greatly appreciated.

Re: Failure starting repl

liamod_1 — Thu, 15 Jun 2023 08:34:25 GMT

Hi @Debayan Mukherjee In the event log I'm seeing an entry saying the Meta-store is down.

DRIVER_HEALTHY 2023-06-15 10:24:21 SAST Driver is healthy.

METASTORE_DOWN 2023-06-15 10:14:20 SAST Metastore is down.

DRIVER_HEALTHY 2023-06-15 10:08:21 SAST Driver is healthy.

RUNNING 2023-06-15 10:08:13 SAST Cluster is running.

INIT_SCRIPTS_FINISHED 2023-06-15 10:07:48 SAST Finished init scripts execution.

INIT_SCRIPTS_STARTED 2023-06-15 10:05:45 SAST Starting init scripts execution.

In root of the traceback I see:

Caused by: java.lang.Throwable: Too many connections

at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:856)

at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:777)

at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:451)

at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1103)

... 159 more

Thanks for reaching out!

Re: Failure starting repl

liamod_1 — Thu, 15 Jun 2023 08:38:57 GMT

@Debayan Mukherjee And at the top of the stack trace:

java.lang.Exception: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

Re: Failure starting repl

liamod_1 — Fri, 16 Jun 2023 10:28:45 GMT

@Debayan Mukherjee I'm unsure so I have messaged our databricks contact. But we have been getting notifications about workspace migrations. Could this be related to discontinued support for ST? Thanks.

Re: Failure starting repl

Aviral-Bhardwaj — Sun, 18 Jun 2023 07:14:19 GMT

most of the time this happen due to Library conflicts please check your library and try to clone that cluster then run

Re: Failure starting repl

Anonymous — Tue, 20 Jun 2023 06:37:25 GMT

Hi @Liam O'Donoghue

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Failure starting repl

liamod_1 — Tue, 20 Jun 2023 09:03:16 GMT

@Aviral Bhardwaj thanks, this seemed to fix the issue, we had an innit script that was potentially conflicting with UI set libraries (in cluster settings).

Re: Failure starting repl

Debayan — Thu, 15 Jun 2023 07:33:31 GMT

Hi, Do you see the errors in event logs? Could you also please check the driver logs for the elaborated error logs and provide it here?

Also, please tag @Debayan with your next comment so I will be getting notified.

Re: Failure starting repl

Debayan — Fri, 16 Jun 2023 06:29:40 GMT

Hi @Liam O'Donoghue , Could you please confirm if you are in ST or E2 shard?