I think it is because you did not attach the libraries to the cluster.

When you work with a notebook, the sparksession is already created.

To add libraries, you should install them on the cluster (in the compute tab) using f.e. pypi/maven etc.