Depending on what you’re doing there is a package called reticulate in R that lets you directly share objects between R and python, Spark not required. https://rstudio.github.io/reticulate/

However I’ve found (so far) it really only works in RStudio, which does run awesomely on Databricks when using the DB ML distributions. You can find RStudio preinstalled on the DB cluster under “apps” on the cluster’s page when you deselect the cluster auto termination for inactivity:

There is a bit more set up once in RStudio on DB to make reticulate work flawlessly that I could post if interested.

What I haven’t tested yet is what happens if you make an Rmd using reticulate in RStudio on Databricks and then try to schedule that in a DB Workflow later. If I do I’ll be sure to post about it in the community.

I’d love it if more direct adoption of reticulate was included in Databricks notebooks that extended it to Scala and SQL too, like you’ve suggested. Each language has its advantages in my opinion, and there are some really awesome ML packages in tidymodels for R that get frequently overlooked by the python community in my opinion because the switching between languages is difficult in most places.