The reason your Volume-based cache isn't working is a credential scoping issue. Databricks only injects UC Volume credentials into init scripts that are themselves stored on a UC Volume. If your init script lives in workspace files or cloud storage, it can't actually read from /Volumes/datalake/test/rlib_cache at execution time — even though the path looks fine and your R code works in a running notebook.

The fix: move your init script to the same Volume (e.g., /Volumes/datalake/test/scripts/init_r.sh). But I'd also change the strategy slightly. Instead of pointing .libPaths() at the Volume path, copy the packages to a local directory during init. Reading libraries over the FUSE mount adds noticeable latency on every library() call.

#!/bin/bash cp -R /Volumes/datalake/test/rlib_cache/* /usr/local/lib/R/site-library/ 2>/dev/null

That copy takes maybe 10-30 seconds for your package set. Way better than 20 minutes of compilation.

Hope this helps! If it helps, mark it as a solution!