Greeting @tariq , this is a great question (thank you @Isi for raising the suggestion). I looked into our internal documentation, and it turns out that it is not recommended to install libraries cluster-wide on "All Purpose" compute in `USER_ISOLATION` mode. Databricks enforces strict separation between users—including how libraries are installed, loaded, and how environment variables are managed.
Key Points to Consider
- USER_ISOLATION clusters strictly restrict cross-user contamination. Init scripts and global environment variables are not always passed into the per-user, per-notebook Python context.
- Cluster-scoped installations (e.g., via init scripts with 'pip install', or through the cluster “Libraries” UI) often do not work as expected in notebook sessions under USER_ISOLATION.
- Instead, per-user '%pip' installs are isolated and recommended.
Recommended Approach
Install within Notebook Sessions
```python
%pip install --extra-index-url <azure-artifact-repo-url> package-name
```
Run this directly in your own notebook. This ensures the package is installed in your user session and correctly respects isolation and credentials.
If You Must Use Cluster Init Scripts
If pre-installation cluster-wide is absolutely necessary, you can test with an init script that explicitly uses the Python executable for notebook sessions. For example:
```bash
#!/bin/bash
/databricks/python/bin/pip install --extra-index-url <azure-artifact-repo-url> <package-name>
```
- Save this script in workspace storage or a mounted volume.
- Add it to the `init_scripts` section of the cluster configuration.
Afterwards, verify installation:
```python
import sys
print(sys.executable)
!pip list
```
Keep in mind: due to `USER_ISOLATION` boundaries, even init scripts may not guarantee availability across all user sessions. Installing with `%pip` inside each notebook is usually more reliable.
In Short
On all-purpose clusters with `USER_ISOLATION`, use `%pip install` with your extra index URL directly in your notebook, and ensure authentication is set for each user session. Init scripts are possible but less reliable for propagating libraries across users.
Hope this helps point you in the right direction!
Cheers, Louis.
