How to use dbx for local development.

[Erik]

​Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing deprecated, and the documentation states :

"Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time."

I am wondering exactly how dbx is supposed to alow local development? At the moment it seems like a job running tool, so enables you to run your whole notebook onaa cluster. But what about interactive development? Attaching a debugger? Is there plans for those features into dbx?

[Hubert-Dudek · Accepted solution]

@Erik Parmann​ , There should soon be a databricks tunnel available which will allow code from IDE to be run directly on databricks (not on spark cluster).

[pichlerpa]

Hi @Hubert Dudek​ , interesting. Do you know about any transparent roadmap or something like this? We are just about to start a new project relying on databricks-connect using RStudio together with sparkly. We were quite surprised about the deprication and not mentioning any real alternative or at least further information. Thanks, Patrick

[-werners-]

@Erik Parmann​ , I was gonna download the latest version of databricks-connect, but then I noticed the following on the help pages (Azure):

Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time.

So I went to the dbx page and noticed:

dbx execute can only be used on clusters with Databricks ML Runtime 7.X and only for Python-based projects.

So I guess scala users have to wait.

[Oliver_Floyd]

Hello @Kaniz Fatma​ ,

We also have just learned that databricks-connect is deprecated and will be replaced by dbx.

This is problematic for us because dbx does not allow to run/debug the code locally.

Can you tell us if it is planned in the near future to reintegrate this functionality.

B.R.

Olivier

[SoniaFoster]

This discussion has piqued my interest, and I'd like to contribute to the conversation by sharing a resource that aligns with our research on effective development practices. This thread is about the intricacies of using Databricks for local development. In addition to this discussion, I'd like to suggest to visit https://limeup.io/blog/sportsbook-software-providers/ a valuable resource. While at first glance this article may seem unrelated, the main takeaway from it is the importance of choosing the right tools and vendors to improve the development experience. Just as choosing the right sportsbook software is critical to the smooth operation of a betting shop, choosing the right tools for local development is essential for effective and efficient coding.

[FeliciaWilliam]

Thank you all for the interesting and useful information

[FeliciaWilliam]

I found answers to my questions here