cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks hiding error stack trace when plotting?

640913
New Contributor III

Hi,

I have been working with data visualisations in Databricks the last week and came across a very peculiar behaviour where it seems like Databricks is actively suppressing errors to the user.

An example:

```

import matplotlib.pyplot as plt

x = [1, 2, 3]

y = [1, 2, 3]

plt.plot(x, y)

x.length

```

produces only the plot as output and I get no information that the code after the plot command has failed. I have attached an image of the output. If I put the `x.length` in its own cell, it will raise the Attribute Error as expected.

Expected behaviour would be that the plot and the Attribute Error is shown. I have attached an image of this when using just ordinary Jupyter Lab.

When reading about Databricks notebooks and errors, I cannot find any information in the documentation that this is expected behaviour. I just searched for "error" but found no indication that Databricks wants to make users aware of this?

  • Have I just configured something incorrectly in my user to create this very unwanted behaviour?
  • If not, why on earth would Databricks do this? Why is this a feature?

I am working in DBR 7.3 LTS ML with Spark 3.0.1 and Scala 2.12 (my company is upgrading the version to DBR 10.4 LTS at the start of 2023).

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions

LandanG
Databricks Employee
Databricks Employee

@gagneriโ€‹ 

So it looks like the error is hidden in DBR 7.3 LTS but it's fixed in DBR 10.4 LTS and later.

On a side note, when it comes to migrate runtimes, I'd recommend checking out the migration guide

View solution in original post

3 REPLIES 3

LandanG
Databricks Employee
Databricks Employee

It might be the DBR version. I ran that code on DBR 11.3 and I got the plot + error. Let me confirm if it's the DBR and get back to you here

LandanG
Databricks Employee
Databricks Employee

@gagneriโ€‹ 

So it looks like the error is hidden in DBR 7.3 LTS but it's fixed in DBR 10.4 LTS and later.

On a side note, when it comes to migrate runtimes, I'd recommend checking out the migration guide

640913
New Contributor III

Alright, thank you.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group