Further, I found that dbutils.library.installPyPI is supported for LTS 5.5 DB version. In my case, I had some PyPI packages which I had installed at cluster level. I removed those cluster level PyPI packages and used dbutils.library.installPyPI to install notebook scoped packages. It works fine now.