uninstalled libraries gets installed back on cluster after restart

ChiragM — Wed, 23 Oct 2024 06:51:51 GMT

I am trying to update libraries on Data bricks cluster by uninstalling existing libraries and installing new ones. When I make api call to uninstall libraries and restart cluster, libraries first show as they are being uninstalled but after cluster comes back into Running state uninstalled libraries gets installed back to the cluster. I have seen similar behavior when I uninstall and restart from Data bricks UI as well. This behavior is intermittent and doesn't happen consistently. I have seen some instances where libraries stay uninstalled.

Notes :

1. There are no init_scripts in the cluster

2. "is_library_for_all_clusters" is set to false when checking for libraries/cluster-status

3. This is happening for both Python and Scala libraries

4. Runtime being used is 12.2LTS

As you can see in the screenshots attached, I have 3 libraries installed initially. I try to uninstall both PyPi libraries and restart. After the restart, we can see that one of the library got installed back on the cluster.

Re: uninstalled libraries gets installed back on cluster after restart

saurabh18cs — Wed, 23 Oct 2024 08:26:38 GMT

maybe you can add some time between uninstalling of libraries and restart

uninstall_libraries()

time.sleep(60) # Wait for the libraries to be uninstalled

restart_cluster()

topic Re: uninstalled libraries gets installed back on cluster after restart in Data Engineering

uninstalled libraries gets installed back on cluster after restart

Re: uninstalled libraries gets installed back on cluster after restart