topic Re: Notebook Visualisations suddenly not working in Data Engineering

Notebook Visualisations suddenly not working

117074 — Fri, 08 Mar 2024 14:01:50 GMT

Hi all,

I have a python script which runs SQL code against our Delta Live Tables and returns a pandas dataframe. I do this multiple times and then use 'display(pandas_dataframe)'. Once this displays I then create a visualization from the UI which is then added to a dashboard.

I've run this script many times with no issues, however in the past 2 hours the behaviour of the notebook seems to have changed. When I run 'display(pandas_dataframe)' it no longer returns in a state that visualisations can be run from it. Instead it returns as if I'm using the print(pandas_dataframe) instead.

Any advice would be great, thank you.

More info'

Using databricks at works through my company cloud set-up. AWS cloud.

Re: Notebook Visualisations suddenly not working

117074 — Fri, 08 Mar 2024 14:15:48 GMT

Update: Upon restarting the cluster the problem seems to have resolved. However, I still need to find out an explanation for why this happened as it has caused additional work to now be done to restore the graphs.

Re: Notebook Visualisations suddenly not working

117074 — Fri, 08 Mar 2024 19:45:17 GMT

Thank you for the detailed response Kaniz, I appreciate it! I do think it may have been cache issues due to there being no spark computation when running them when the error occured.

It did lead me down a train of thought.. is it possible to extract the code used to generate the graphs from the visualisation UI? That way instead of doing display(pandas_df) and manually making the visualisations myself - I can just take this code. Alternatively, I may look into using some plotly graphs!

Thank you