Hi experts,

Does anyone know if there's a way of properly debugging python code outside of notebooks?

We have a complicated python-based framework for loading files, transforming them according to the business specification and saving the results into delta tables. The whole framework is structured as a python project with multiple folders, .py files and classes inside of these files. In order to perform a proper debugging (with breakpoints, conditional breakpoints, stepping into methods and so on) I had to set up a local development environment on my Windows machine that mimics our cloud setup (local directories serving as Azure storage containers, delta tables created locally and so on).

This setup works, but it's not really convenient. I was wondering if there's a way of performing advanced debugging in Databricks either remotely from my IDE (VSCode) or via Databricks UI?

After searching the forum I found no definitive answer to this question. Notebook debugger is not an option because it's meant for notebooks only. %debug command and pdb are inconsistent in terms of their behavior. Databricks Connect is something I can't test at work because of our security setup (too restrictive).

Best regards,
Vladislavs