pdb debugger on databricks

johnp — Mon, 18 Mar 2024 22:41:25 GMT

I am new to databricks. and trying to debug my python application with variable-explore by following the instruction from: https://www.databricks.com/blog/new-debugging-features-databricks-notebooks-variable-explorer

I added the "import pdb" in the first cell, then "pdb.set_trace()" inside the functions I wanted to debug. After running all the cells, the databricks seems stopped at the pdb.set_trace(), but I did not get the cell for ipdb prompt. Also the list of variables does not updated ( the local variables did not show on the right side of panel). Is there something I missing here?

From another help page: https://docs.databricks.com/en/_extras/notebooks/source/python-debugger.html, just wonder if I need to set %debug or %pdb instead of %python to get ipdb prompt. But this page seems quite old, not sure it is still valid.

Re: pdb debugger on databricks

johnp — Sat, 23 Mar 2024 15:51:32 GMT

I test with some simple applications, it works as you described. However, the application I am debugging uses the pyspark structured streaming, which runs continuously. After inserting pdb.set_trace(), the application paused at the breakpoint, but the application continues to run for incoming stream data, and no ipdb prompt shows up. How do I get the ipdb> prompt in this case?

topic pdb debugger on databricks in Machine Learning

pdb debugger on databricks

Re: pdb debugger on databricks