Databricks hiding error stack trace when plotting?

640913 — Mon, 14 Nov 2022 12:19:06 GMT

Hi,

I have been working with data visualisations in Databricks the last week and came across a very peculiar behaviour where it seems like Databricks is actively suppressing errors to the user.

An example:

```

import matplotlib.pyplot as plt

x = [1, 2, 3]

y = [1, 2, 3]

plt.plot(x, y)

x.length

```

produces only the plot as output and I get no information that the code after the plot command has failed. I have attached an image of the output. If I put the `x.length` in its own cell, it will raise the Attribute Error as expected.

Expected behaviour would be that the plot and the Attribute Error is shown. I have attached an image of this when using just ordinary Jupyter Lab.

When reading about Databricks notebooks and errors, I cannot find any information in the documentation that this is expected behaviour. I just searched for "error" but found no indication that Databricks wants to make users aware of this?

Have I just configured something incorrectly in my user to create this very unwanted behaviour?
If not, why on earth would Databricks do this? Why is this a feature?

I am working in DBR 7.3 LTS ML with Spark 3.0.1 and Scala 2.12 (my company is upgrading the version to DBR 10.4 LTS at the start of 2023).

Thank you.

Re: Databricks hiding error stack trace when plotting?

LandanG — Mon, 14 Nov 2022 18:26:22 GMT

It might be the DBR version. I ran that code on DBR 11.3 and I got the plot + error. Let me confirm if it's the DBR and get back to you here

Re: Databricks hiding error stack trace when plotting?

LandanG — Mon, 14 Nov 2022 18:36:19 GMT

@gagneri

So it looks like the error is hidden in DBR 7.3 LTS but it's fixed in DBR 10.4 LTS and later.

On a side note, when it comes to migrate runtimes, I'd recommend checking out the migration guide

Re: Databricks hiding error stack trace when plotting?

640913 — Tue, 22 Nov 2022 09:45:07 GMT

Alright, thank you.

topic Re: Databricks hiding error stack trace when plotting? in Data Engineering

Databricks hiding error stack trace when plotting?

Re: Databricks hiding error stack trace when plotting?

Re: Databricks hiding error stack trace when plotting?

Re: Databricks hiding error stack trace when plotting?