Debugging python code outside of Notebooks

[vvzadvor]

Hi experts,

Does anyone know if there's a way of properly debugging python code outside of notebooks?

We have a complicated python-based framework for loading files, transforming them according to the business specification and saving the results into delta tables. The whole framework is structured as a python project with multiple folders, .py files and classes inside of these files. In order to perform a proper debugging (with breakpoints, conditional breakpoints, stepping into methods and so on) I had to set up a local development environment on my Windows machine that mimics our cloud setup (local directories serving as Azure storage containers, delta tables created locally and so on).

This setup works, but it's not really convenient. I was wondering if there's a way of performing advanced debugging in Databricks either remotely from my IDE (VSCode) or via Databricks UI?

After searching the forum I found no definitive answer to this question. Notebook debugger is not an option because it's meant for notebooks only. %debug command and pdb are inconsistent in terms of their behavior. Databricks Connect is something I can't test at work because of our security setup (too restrictive).

Best regards,
Vladislavs

[szymon_dybczak · Accepted solution]

Other than databricks connect, I'm afraid there isn't much option there. 

You can also try to setup your own apache spark environment with docker, but any commands specific to datbricks will not work. 

[vvzadvor]

Thank you for the response. Have you tried remote debugging with Databricks Connect V2? I am particularly interested in stepping into the methods from the libraries installed on the cluster. Is it possible?

As for setting up a local Spark environment with Docker - it is even more complicated in my company due to security restrictions. The setup I mentioned in my post was the best thing I could achieve given the limitations.

[vvzadvor]

I actually managed to get a permission for using pat token to connect to databricks development environment. And with that I managed to set up VSCode extension for databricks, connect to cluster and create sync location. I can even run spark apps from my local machine on a databricks cluster. However, whenever I am trying to debug the code, debugger simply skips the breakpoints.

[vvzadvor]

OK, I can now confirm that remote debugging with stepping into your own libraries installed on the cluster is possible and is actually pretty convenient using a combination of databricks-connect Python library and a Databricks extension for VSCode. Setting it up is pretty straight forward as well (everything is documented). Only one part confused me: all your Python code runs locally and only spark code is sent to the cluster. It means that your launch.json might end up looking like this (see below) and you actually need to use a profile related to debugpy (with justMyCode set to false) when initiating debugging session. Hopefully, it helps someone.