display() not updating after 1000 rows

[MrsBaker]

Hello folks! I am calling display() on a streaming query sourced from a delta table. The output from display() displays the new rows added to the source table. But as soon as the output results hit 1000 rows, the output is not updated anymore. As a result, the visualization I built on top of the output is stale. Is there a work around for the display function to append the new results while clearing the old results after the display limit of 1000 is hit?

[MrsBaker]

aggregate function followed by timestamp field sorted in descending order did the trick:

streaming_df.groupBy("field1", "time_field").max("field2").orderBy(col("time_field").desc()).display()