cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to use dbx for local development.

Erik
Valued Contributor II

​Databricks connect is a program which allows you to run spark code locally, but the actual execution happens on a spark cluster. Noticeably, it allows you to debug and step through the code locally in your own IDE. Quite useful. But it is now beeing deprecated, and the documentation states :

"Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time."

I am wondering exactly how dbx is supposed to alow local development? At the moment it seems like a job running tool, so enables you to run your whole notebook onaa cluster. But what about interactive development? Attaching a debugger? Is there plans for those features into dbx?

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

@Erik Parmann​ , There should soon be a databricks tunnel available which will allow code from IDE to be run directly on databricks (not on spark cluster).

View solution in original post

7 REPLIES 7

Hubert-Dudek
Esteemed Contributor III

@Erik Parmann​ , There should soon be a databricks tunnel available which will allow code from IDE to be run directly on databricks (not on spark cluster).

Hi @Hubert Dudek​ , interesting. Do you know about any transparent roadmap or something like this? We are just about to start a new project relying on databricks-connect using RStudio together with sparkly. We were quite surprised about the deprication and not mentioning any real alternative or at least further information. Thanks, Patrick

-werners-
Esteemed Contributor III

@Erik Parmann​ , I was gonna download the latest version of databricks-connect, but then I noticed the following on the help pages (Azure):

Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time.

So I went to the dbx page and noticed:

dbx execute can only be used on clusters with Databricks ML Runtime 7.X and only for Python-based projects.

So I guess scala users have to wait.

Hello @Kaniz Fatma​ ,

We also have just learned that databricks-connect is deprecated and will be replaced by dbx.

This is problematic for us because dbx does not allow to run/debug the code locally.

Can you tell us if it is planned in the near future to reintegrate this functionality.

B.R.

Olivier

SoniaFoster
New Contributor II

This discussion has piqued my interest, and I'd like to contribute to the conversation by sharing a resource that aligns with our research on effective development practices. This thread is about the intricacies of using Databricks for local development. In addition to this discussion, I'd like to suggest to visit https://limeup.io/blog/sportsbook-software-providers/ a valuable resource. While at first glance this article may seem unrelated, the main takeaway from it is the importance of choosing the right tools and vendors to improve the development experience. Just as choosing the right sportsbook software is critical to the smooth operation of a betting shop, choosing the right tools for local development is essential for effective and efficient coding.

FeliciaWilliam
Contributor

Thank you all for the interesting and useful information

FeliciaWilliam
Contributor

I found answers to my questions here

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group