This seems to solve the problem: https://github.com/sparklyr/sparklyr/issues/3449Apparently sparklyr requires that Unity Catalog is enabled on the cluster in order to get the connection working right.
Having the same issue when trying to connect Rstudio to Databricks Cluster. Connection seems to be fine until the end error gets thrown: Error in if (grepl("UNAVAILABLE", status_error)) { :
argument is of length zeroDid you solve this problem?