04-03-2022 07:53 AM
Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing deprecated, and the documentation states :
"Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time."
I am wondering exactly how dbx is supposed to alow local development? At the moment it seems like a job running tool, so enables you to run your whole notebook onaa cluster. But what about interactive development? Attaching a debugger? Is there plans for those features into dbx?
04-03-2022 08:26 AM
@Erik Parmann , There should soon be a databricks tunnel available which will allow code from IDE to be run directly on databricks (not on spark cluster).
04-03-2022 08:26 AM
@Erik Parmann , There should soon be a databricks tunnel available which will allow code from IDE to be run directly on databricks (not on spark cluster).
04-24-2022 10:43 PM
Hi @Hubert Dudek , interesting. Do you know about any transparent roadmap or something like this? We are just about to start a new project relying on databricks-connect using RStudio together with sparkly. We were quite surprised about the deprication and not mentioning any real alternative or at least further information. Thanks, Patrick
04-05-2022 01:59 AM
@Erik Parmann , I was gonna download the latest version of databricks-connect, but then I noticed the following on the help pages (Azure):
Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time.
So I went to the dbx page and noticed:
dbx execute can only be used on clusters with Databricks ML Runtime 7.X and only for Python-based projects.
So I guess scala users have to wait.
11-16-2022 02:35 AM
Hello @Kaniz Fatma ,
We also have just learned that databricks-connect is deprecated and will be replaced by dbx.
This is problematic for us because dbx does not allow to run/debug the code locally.
Can you tell us if it is planned in the near future to reintegrate this functionality.
B.R.
Olivier
03-09-2024 02:02 PM
This discussion has piqued my interest, and I'd like to contribute to the conversation by sharing a resource that aligns with our research on effective development practices. This thread is about the intricacies of using Databricks for local development. In addition to this discussion, I'd like to suggest to visit https://limeup.io/blog/sportsbook-software-providers/ a valuable resource. While at first glance this article may seem unrelated, the main takeaway from it is the importance of choosing the right tools and vendors to improve the development experience. Just as choosing the right sportsbook software is critical to the smooth operation of a betting shop, choosing the right tools for local development is essential for effective and efficient coding.
05-24-2024 06:15 AM
Thank you all for the interesting and useful information
06-03-2024 01:46 AM
I found answers to my questions here
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group