Hello. I'm the current maintainer of sparklyr (a R interface for Apache Spark) and a few sparklyr extensions such as sparklyr.flint.

Sparklyr was fortunate to receive some contribution from Databricks folks, which enabled R users to run `spark_connect(method = "databricks")` to connect to Databricks Runtime.

My question is how to make this type of Spark connection in R work with sparklyr extensions (e.g., see https://github.com/r-spark/sparklyr.flint/issues/55 -- this was something I don't have a good answer for at the moment because I'm not really super-familiar with how Databricks connections work with sparklyr internally).

A bit more context: sparklyr.flint is an R interface for the Flint time series library that works on top of sparklyr. Usually when users run code such as the following

library(sparklyr)
library(sparklyr.flint)
 
sc <- spark_connect(master = "yarn-client", spark_home = "/usr/lib/spark")

The presence of sparklyr.flint as a sparklyr extension will cause the Spark process to fetch some version of Flint time series library jar files and load those files within the Spark session that it is connecting to.

But this didn't work if we were to replace the `sc <- spark_connect(...)` from above with `sc <- spark_connect(method = "databricks")` (again, see https://github.com/r-spark/sparklyr.flint/issues/55 for details). My uneducated guess is `method = "databricks"` had some level of indirection involved in the connecting-to-Spark step, and the Flint time series jar files were downloaded into the wrong location.

I'm wondering whether there is some simple change to sparklyr I can make to ensure sparklyr extensions also work in Databricks. Your input would be greatly appreciated.

Thanks!