pdb debugger on databricks

[johnp]

I am new to databricks. and trying to debug my python application with variable-explore by following the instruction from: https://www.databricks.com/blog/new-debugging-features-databricks-notebooks-variable-explorer

I added the "import pdb" in the first cell, then "pdb.set_trace()"  inside the functions I wanted to debug. After running all the cells,  the databricks seems stopped at the pdb.set_trace(), but I did not get the cell for ipdb prompt.  Also the list of variables does not updated ( the local variables did not show on the right side of panel).   Is there something I missing here?

From another help page: https://docs.databricks.com/en/_extras/notebooks/source/python-debugger.html, just wonder if I need to set %debug or %pdb instead of %python to get ipdb prompt. But this page seems quite old, not sure it is still valid.

[johnp]

I test with some simple applications, it works as you described.  However, the application I am debugging uses the pyspark structured streaming, which runs continuously. After inserting pdb.set_trace(), the application paused at the breakpoint, but the application continues to run for incoming stream data, and no ipdb prompt shows up.  How do I get the ipdb> prompt in this case?