cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks hiding error stack trace when plotting?

640913
New Contributor III

Hi,

I have been working with data visualisations in Databricks the last week and came across a very peculiar behaviour where it seems like Databricks is actively suppressing errors to the user.

An example:

```

import matplotlib.pyplot as plt

x = [1, 2, 3]

y = [1, 2, 3]

plt.plot(x, y)

x.length

```

produces only the plot as output and I get no information that the code after the plot command has failed. I have attached an image of the output. If I put the `x.length` in its own cell, it will raise the Attribute Error as expected.

Expected behaviour would be that the plot and the Attribute Error is shown. I have attached an image of this when using just ordinary Jupyter Lab.

When reading about Databricks notebooks and errors, I cannot find any information in the documentation that this is expected behaviour. I just searched for "error" but found no indication that Databricks wants to make users aware of this?

  • Have I just configured something incorrectly in my user to create this very unwanted behaviour?
  • If not, why on earth would Databricks do this? Why is this a feature?

I am working in DBR 7.3 LTS ML with Spark 3.0.1 and Scala 2.12 (my company is upgrading the version to DBR 10.4 LTS at the start of 2023).

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions

LandanG
Honored Contributor
Honored Contributor

@gagneri​ 

So it looks like the error is hidden in DBR 7.3 LTS but it's fixed in DBR 10.4 LTS and later.

On a side note, when it comes to migrate runtimes, I'd recommend checking out the migration guide

View solution in original post

3 REPLIES 3

LandanG
Honored Contributor
Honored Contributor

It might be the DBR version. I ran that code on DBR 11.3 and I got the plot + error. Let me confirm if it's the DBR and get back to you here

LandanG
Honored Contributor
Honored Contributor

@gagneri​ 

So it looks like the error is hidden in DBR 7.3 LTS but it's fixed in DBR 10.4 LTS and later.

On a side note, when it comes to migrate runtimes, I'd recommend checking out the migration guide

640913
New Contributor III

Alright, thank you.