You’re seeing two key issues with your RStudio Server on Azure Databricks:

1. RStudio stops working after 1–2 days.

2. You get permission errors using sparklyr and can’t update the Connections pane.

Let’s address each:

1. RStudio Server Stops Working After 1–2 Days

This often happens due to how Databricks clusters work:

• Clusters are Ephemeral: When a cluster is terminated or restarted (either by policy, idle timeout, or an admin), all software and changes not baked into the cluster’s custom image are lost. Your manual RStudio installation via notebooks is not persistent.

• Manual Installs are Non-Persistent: When the cluster spins down, the next startup will run on a fresh VM, losing your RStudio install.

How to Fix
You need to make RStudio part of the cluster’s initialization so it’s always present:

• Use an Init Script:
  Place a shell script (like your install commands) on DBFS or workspace, then configure your cluster to run this at startup.

  • This ensures RStudio and its dependencies are re-installed every time the cluster starts, so it’s always present.

Example Init Script Steps:

1. Move your install commands into a .sh script file.

2. Place the script in DBFS (e.g., /dbfs/FileStore/scripts/rstudio-install.sh).

3. In the cluster configuration, set the script as a cluster init script.

Reference:
Databricks Init Scripts:
https://docs.databricks.com/en/clusters/init-scripts.html

2. sparklyr Permission Errors (“Permission denied” on .jar files)

You see:

• cannot create file ‘/usr/local/lib/R/site-library/sparklyr/java//sparklyr-2.2-2.11.jar’, reason 'Permission denied'

This is due to how you’re installing R packages and user permissions:

• Default Library Path is Root-Owned: On Databricks, /usr/local/lib/R/site-library is typically owned by root, but RStudio users run as a non-root user.

• sparklyr Tries to Write .jar Files: When you load sparklyr, it tries to write required Java/JAR files to its library path and fails due to lack of permission.

• Solution is to Use User Library Paths: Install R packages in a user writeable location and run R sessions with this as the library path.

How to Fix

• Change R Library Path:
  In your R session or RStudio, set the library path to a writeable directory. For example:

• Make sure this directory exists and is writeable.
  Alternatively, install R packages using:

• Do Not Install R Packages as Root via shell or sudo unless you are sure all users run as root (not recommended on Databricks).

• Reference:

  • Databricks R package management best practices

3. sparklyr Connection Pane Error

This is expected when using sparklyr with Databricks, as the "Connections" feature in RStudio might not update correctly under gateway/proxy setups. As the warning suggests, you can suppress these with:

But this doesn't affect actual Spark functionality—it only impacts the UI pane.

Summary Table