@Retired_mod,

I've been running some additional experiments.

Experiment 1

1. Create DBR 12.2 cluster
2. Once running, check to see if /home/libraries/.cache/pip exists using ls -lh (it does not)
3. Use cluster configuration "Libraries" tab to install "s3_ingest==0.0.2" (success)
4. Check again to see if pip cache folder exists (it does not)
5. Use cluster configuration "Libraries" tab to uninstall "s3_ingest==0.0.2" (pending restart)

Experiment 2

1. Edit the cluster configuration from "Experiment 1", change to DBR 13.2 then click "Confirm and Restart"
2. Once running, check to see if /home/libraries/.cache/pip exists using ls -lh (it does not)
3. Run /databricks/python/bin/python -m pip uninstall -y s3_ingest to verify s3_ingest is not installed (returns warning that package is not installed)
4. Use cluster configuration "Libraries" tab to install "s3_ingest==0.0.2" (failure, same error as OP)

Experiment 3

1. Uninstall s3_ingest==0.0.2 after "Experiment 2" (pending restart), then restart the cluster
2. Once restart completes, run mkdir -p /home/libraries/.cache/pip then chown -R nobody:nogroup /home/libraries.
3. Use cluster configuration "Libraries" tab to install "s3_ingest==0.0.2" (failure, same error as OP)

Conclusion

This seems like a bug to me. I'm going to look into running pip install from an init script as a workaround but the "Libraries" functionality seems broken to me - both for interactive and job compute. As I've verified, pip does not require sudo or sudo -H, it runs just fine on it's own. Regardless, the "Libraries" functionality does not provide the ability to specify sudo which means I can't use the feature and I likely need to create a pip install script for every job instead of using the built-in "libraries" functionality with "pypi".