topic Using the %debug magic in DataBricks notebook in Data Engineering

Using the %debug magic in DataBricks notebook

327753 — Wed, 26 Oct 2022 09:39:26 GMT

When developing locally, I can write %debug in a new cell after encountering an error, and jump into the function that the error originated from. In Databricks, this freezes the notebook indefinitely.

For example:

In [1]:

def query_data():

df_full = query("SELECT * FROM mytable")

df = df_full[df_full.mycolumn > 100]

assert not df.empty

return df

query_data()

Out [1]: AssertionError

In [2] %debug

(interactive prompt to explore df_full)

How do I get this functionality? I can get it in the Cloud Terminal, but that doesn't have the same python environment as my notebooks, so I can't really test things.

Re: Using the %debug magic in DataBricks notebook

327753 — Mon, 31 Oct 2022 09:37:31 GMT

If I am reading my "compute" tab correctly we are using 10.4 To my knowledge, we have no reason not to upgrade, and now it looks like pdb would be a solid benefit. Any warnings before I try that upgrade?

Re: Using the %debug magic in DataBricks notebook

327753 — Mon, 31 Oct 2022 10:11:27 GMT

I just upgraded my personal node and %debug worked! I appreciate the reminder to use pdb() itself when appropriate too.

I'm still interested in whether we should have any concerns about upgrading our main cluster - please do let me know, and then I'll happily close out the question!