I am also looking for a way to debug code with local IDE based development. We have a complex and mature ML codebase of python code (.py files, not notebooks) and use debugging extensively (vscode). It is critical to be able to step through code and inspect variables etc. This simply is not possible with notebooks.

We are looking at migrating this codebase onto Databricks but we have concerns regarding IDE based development and debugging capabilities. We want to avoid the situation where we sacrifice software best practices when using Databricks.

While notebook-based development seems suitable for the Databricks example ML projects, which use existing NN architectures / feature extraction / datasets, I would argue that this type of development is almost impossible for a large team of ML researchers / engineers contributing to a mature ML codebase.

I am also confused by the recommended tooling for IDE based development. databricks-connect is not recommended (deprecated?) and it is recommended that dbx is used instead. Dbx does not offer any debugging capabilities. It also forces you to use either notebooks or python packages. Ideally we want to execute the codebase as is from the IDE on a compute cluster with debugging.

Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. See: https://docs.databricks.com/dev-tools/databricks-connect.html#databricks-connect

dbx currently doesn't provide interactive debugging capabilities. If you want to use interactive debugging, you can use Databricks Connect, and then use dbx for deployment operations. See https://dbx.readthedocs.io/en/latest/intro/?h=debug#limitations

It seems that databricks-connect is the closest option for debugging within an IDE. However, from what I understand, only the spark related commands are sent to the Databricks compute cluster while the rest of the code is executed on the local machine. (See limitations https://docs.databricks.com/dev-tools/databricks-connect.html#limitations). This hybrid approach would be challenging to develop with as the majority of our code does not use spark APIs, and there is potential environment mismatch.

Finally, there is the vscode Databricks extension https://github.com/paiqo/Databricks-VSCode. I believe this allows you to execute notebooks on compute clusters. I have tried using this and can not find any support for debugging.

Any recommendations would be greatly appreciated!

Thanks