Notebook Visualisations suddenly not working
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2024 06:01 AM
Hi all,
I have a python script which runs SQL code against our Delta Live Tables and returns a pandas dataframe. I do this multiple times and then use 'display(pandas_dataframe)'. Once this displays I then create a visualization from the UI which is then added to a dashboard.
I've run this script many times with no issues, however in the past 2 hours the behaviour of the notebook seems to have changed. When I run 'display(pandas_dataframe)' it no longer returns in a state that visualisations can be run from it. Instead it returns as if I'm using the print(pandas_dataframe) instead.
Any advice would be great, thank you.
More info'
Using databricks at works through my company cloud set-up. AWS cloud.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2024 06:15 AM
Update: Upon restarting the cluster the problem seems to have resolved. However, I still need to find out an explanation for why this happened as it has caused additional work to now be done to restore the graphs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2024 11:45 AM
Thank you for the detailed response Kaniz, I appreciate it! I do think it may have been cache issues due to there being no spark computation when running them when the error occured.
It did lead me down a train of thought.. is it possible to extract the code used to generate the graphs from the visualisation UI? That way instead of doing display(pandas_df) and manually making the visualisations myself - I can just take this code. Alternatively, I may look into using some plotly graphs!
Thank you
