@Priya K​ :

When you install a custom library using the install_libraries API in Databricks, the installed version is cached on the worker nodes until the cluster is restarted. Uninstalling the library will remove it from the cache, but it will still be loaded in memory by any running tasks until the cluster is restarted.

To force the cluster to use the newly installed version of the library, you can try a few options:

1. Use a new cluster: You can create a new cluster and install the new version of the library on that cluster. This way, you can ensure that the new version of the library is used without any conflicts with the old version.
2. Reload the module: If you have already installed the new version of the library and want to start using it in your code without restarting the cluster, you can try reloading the module using importlib.reload(). This will reload the module and use the new version of the library.
3. Restart the Python interpreter: You can also try restarting the Python interpreter by using the os.execv() function. This will restart the Python interpreter and load the new version of the library.

However, keep in mind that these options may have some limitations and risks depending on your use case. For example, reloading the module or restarting the Python interpreter may cause some inconsistencies or conflicts with the existing tasks running on the cluster. Additionally, creating a new cluster may increase your overall costs and may not be feasible for all use cases.