Hi,
I'm experiencing an issue with a Databricks notebook after restarting a cluster that has a custom library installed.
Here are the steps I followed:
- Created a Databricks compute cluster.
- Attached a notebook to the cluster.
- Installed a library required to connect to Hive.
- Restarted the cluster.
- Ran the Hive connection code in the notebook, and it executed successfully.
After a few hours, the cluster auto-terminated.
The next day, I restarted the same cluster and opened the same notebook (still attached to that cluster). However, when I tried to run even simple commands, such as a basic SQL query or print statement, the commands would remain in a running state indefinitely and never complete.
The only workaround I've found is to create a new cluster, install the library again, and repeat the setup process. Everything works correctly until the cluster terminates again.
This behavior seems to occur whenever I install a library on a cluster. The notebook works fine initially, but after the cluster is terminated and restarted, commands hang and do not execute.
Is this expected behavior? Has anyone encountered a similar issue or knows what might be causing it?
Thanks in advance for your help.
