Best way to make a scrollable DataFrame in a ipywidgets.Output?

[yhyhy3]

I have a pretty complex Jupyter widgets UI in a databricks notebook that has a dataframe that

1. will be modified by some Jupyter widget callbacks

2. needs to be displayed to the user and updated as it is modified

3. is large and needs to support vertical scrolling

Since callbacks cannot directly access cell output, I have to use a widget (either ipywidgets.Output or ipywidgets.HTML) to display the dataframe.

Here are the things I tried (finally succeeding with the 4th attempt/workaround):

# Setup
import ipywidgets as widgets
import pandas as pd
pd.set_option("display.max_rows", 1000)
df = pd.DataFrame([[1, 2]] * 50)
 
# Attempt 1 - Use widgets.Output with databricks dataframe formatting
a1 = widgets.Output(layout={"scroll": "auto", "height": "300px"})
with a1:
    display(df)
display(a1)
##### RESULT ######
# <Databricks Output (not supported in output widgets)>
################
 
# Attempt 2 - Use widgets.Output with pandas dataframe formatting
from IPython.display import HTML
a2 = widgets.Output(layout={"scrollY": "auto", "height": "300px"})
with a2:
    display(HTML(df.to_html()))
display(a2)
##### RESULT ######
# Non-Databricks formatted pandas DataFrame but does not limit the height
# and there is no scrolling
################
 
# Attempt 3 - Use widgets.HTML with pandas dataframe formatting
from IPython.display import HTML
a3 = widgets.HTML(layout={"scroll": "auto", "height": "100px"})
a3.value = df.to_html()
display(a3)
##### RESULT ######
# Non-Databricks formatted pandas DataFrame and limit the height,
# but there is no scrolling, so the dataframe is truncated to a few rows.
################
 
# Attempt 4 - Use widgets.Output with pandas dataframe formatting and
#                       wrap it in a widgets.VBox
a4 = widgets.Output()
 
with a4:
    display(HTML(df.to_html()))
display(widgets.VBox(children=[a4], layout={"height": "300px", "scroll": "auto"}))
##### RESULT ######
# SUCCESS!!
# Non-Databricks formatted pandas DataFrame with scrolling enabled
################

Clearly, using widgets.VBox wrapping widgets.Output with a workaround to use the pandas to_html instead of Databrick's to_html function is not ideal. Is there a better way to make a scrollable DataFrame in a Databricks notebook? Or is this the best workaround until "<Databricks Output (not supported in output widgets)>" is fixed?

[Kaniz]

Hi @Yushi Homma​,

Based on the code provided, it seems that you have already explored multiple options for displaying a scrollable DataFrame in ipywidgets.Output widget. The workaround you found using widgets.Output wrapped in widgets.VBox with pandas to_html formatting is indeed a viable solution.

However, if you are specifically looking for a solution that is compatible with Databricks notebooks and does not require workarounds, it's worth noting that the "<Databricks Output (not supported in output widgets)>" message indicates that Databricks-specific output is not directly supported in ipywidgets.Output.

In this case, until Databricks resolves this limitation, the workaround you found may be the best available option.

Alternatively, you could consider using other visualization libraries or interactive DataFrame widgets that are compatible with Databricks notebooks.

For example, you can explore libraries such as Sparklyr or Koalas which provide interactive DataFrame visualizations within Databricks. These libraries are specifically designed for working with large datasets and may offer built-in support for scrollable DataFrames.

[Anonymous]

Hi @Yushi Homma​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!