Using the %debug magic in DataBricks notebook

[327753]

When developing locally, I can write %debug in a new cell after encountering an error, and jump into the function that the error originated from. In Databricks, this freezes the notebook indefinitely.

For example:

In [1]:

def query_data():

df_full = query("SELECT * FROM mytable")

df = df_full[df_full.mycolumn > 100]

assert not df.empty

return df

query_data()

Out [1]: AssertionError

In [2] %debug

(interactive prompt to explore df_full)

How do I get this functionality? I can get it in the Cloud Terminal, but that doesn't have the same python environment as my notebooks, so I can't really test things.

[Kaniz · Accepted solution]

Hi @Matt Ritter​, Notebooks run on Databricks Runtime 11.2 and above only support The Python Debugger (pdb). Which DBR version are you using?

[327753]

If I am reading my "compute" tab correctly we are using 10.4 To my knowledge, we have no reason not to upgrade, and now it looks like pdb would be a solid benefit. Any warnings before I try that upgrade?​

[327753]

I just upgraded my personal node and %debug worked! I appreciate the reminder to use pdb() itself when appropriate too.

I'm still interested in whether we should have any concerns about upgrading our main cluster - please do let me know, and then I'll happily close out the question!

[Kaniz]

Hi @Matt Ritter​, The following release notes provide information about Databricks Runtime 11.2, powered by Apache Spark™ 3.3.0. Databricks released these images in September 2022.