Notebook commands hang after restarting a cluster with custom library installed

DineshOjha
New Contributor III

3 weeks ago

Hi,

I'm experiencing an issue with a Databricks notebook after restarting a cluster that has a custom library installed.

Here are the steps I followed:

  1. Created a Databricks compute cluster.
  2. Attached a notebook to the cluster.
  3. Installed a library required to connect to Hive.
  4. Restarted the cluster.
  5. Ran the Hive connection code in the notebook, and it executed successfully.

After a few hours, the cluster auto-terminated.

The next day, I restarted the same cluster and opened the same notebook (still attached to that cluster). However, when I tried to run even simple commands, such as a basic SQL query or print statement, the commands would remain in a running state indefinitely and never complete.

The only workaround I've found is to create a new cluster, install the library again, and repeat the setup process. Everything works correctly until the cluster terminates again.

This behavior seems to occur whenever I install a library on a cluster. The notebook works fine initially, but after the cluster is terminated and restarted, commands hang and do not execute.

Is this expected behavior? Has anyone encountered a similar issue or knows what might be causing it?

Thanks in advance for your help.

balajij8
Esteemed Contributor II

3 weeks ago

@DineshOjha 

The issue here is likely tied to the Spark or Python REPL hanging during its initialization phase on boot. When custom libraries are installed - the packaging Hive clients, custom Hadoop configurations or conflicting JARs -dependency conflicts can break the execution context. If a library overwrites a core runtime dependency or introduces a class path collision upon startup, the REPL cannot reach an interactive state, causing any submitted command. The queries will get queued indefinitely or eventually get cancelled.

Hive metastore connectivity issues are another frequent cause of REPL initialization hanging indefinitely. If the library or its associated configuration triggers an external metastore connection upon session creation, network latency, misconfigured endpoints or credential timeouts can block the driver from finishing its startup sequence without failing fast.

You can check the driver logs directly on the cluster's configuration page under the Driver Logs section (log4j and standard error logs) while the notebook is hanging. The captured stack traces will show the thread, class path collision or blocked network call preventing the REPL from completing initialization. You can use lake house federation instead of custom libraries to connect to hive meta store if feasible. More details here

aayush_410
New Contributor II

3 weeks ago

Cluster-scoped libraries aren't a one-time install — they get reinstalled automatically every time the cluster restarts. If that reinstall stalls (network path to the library source, private repo/artifactory unreachable, dependency conflict with the Hive connector), the driver's REPL never finishes initializing, so even print() or a trivial SQL cell just sits in "Running" forever, waiting behind the pending library install.

Move the Hive connector library to a cluster-scoped init script instead of the UI "Install Library" button. Init scripts run before the Python REPL comes up and are far more reliable/deterministic than the async library-install mechanism, especially for connector libraries with native/JAR dependencies.

Aayush Sharma

DineshOjha
New Contributor III

2 weeks ago

Thank you for the response. From the driver logs I gather that the REPL has crashed, but this isn't the case only with Hive. I have noticed this pattern with Oracle library installation as well.

0 Kudos