Databricks

J15S · ‎12-21-2022

Is anybody actually using the RStudio app integration on Databricks? I'm surprised to find so little discussion in this forum.

My team has been using it for about 3 months and it seems under-developed.

1) No automated backup, you have to do it yourself Rstudio-automated-backup - Databricks

2) Spark connection gets dropped on a regular basis

3) Can't access repos in the `/Workspaces/Repos/` path though I can see it there on the web terminal.

Is everyone basically using notebooks?

jonathan-dufaul · ‎12-23-2022

I tried it but I too felt it was clunky for the above reasons (3 being the fatal one). In addition:

can't use it on a cluster that autoterminates/actually had to have a dedicated cluster for each user I believe
the login process for the open source version is bad (allowing one user and having to share the credentials)
persisting the workspace at all is a pain

All of this work is to avoid using a notebook that is tightly integrated with databricks. The reward isn't worth the effort.

J15S · ‎01-03-2023

@Jonathan Dufault Thanks for the response, and glad I'm not alone. My problem (and this is probably just a preference thing) is that the 'reward' of using a full-fledged IDE is huge, compared to bouncing between notebooks in multiple tabs. The integrations are all (mostly?) Databricks specific and my team hasn't fully committed yet to try and keep some platform independence so we're not getting much benefit there.