https://community.databricks.com/t5/data-engineering/display-not-updating-after-1000-rows/m-p/35041#M25726 <P>Hello folks! I am calling&nbsp;display()&nbsp;on a streaming query sourced from a delta table. The output from&nbsp;display()&nbsp;displays the new rows added to the source table. But as soon as the output results hit 1000 rows, the output is not updated anymore. As a result, the visualization I built on top of the output is stale. Is there a work around for the display function to append the new results while clearing the old results after the display limit of 1000 is hit?</P> Mon, 08 Aug 2022 17:53:57 GMT MrsBaker 2022-08-08T17:53:57Z https://community.databricks.com/t5/data-engineering/display-not-updating-after-1000-rows/m-p/35041#M25726 <P>Hello folks! I am calling&nbsp;display()&nbsp;on a streaming query sourced from a delta table. The output from&nbsp;display()&nbsp;displays the new rows added to the source table. But as soon as the output results hit 1000 rows, the output is not updated anymore. As a result, the visualization I built on top of the output is stale. Is there a work around for the display function to append the new results while clearing the old results after the display limit of 1000 is hit?</P> Mon, 08 Aug 2022 17:53:57 GMT https://community.databricks.com/t5/data-engineering/display-not-updating-after-1000-rows/m-p/35041#M25726 MrsBaker 2022-08-08T17:53:57Z https://community.databricks.com/t5/data-engineering/display-not-updating-after-1000-rows/m-p/35042#M25727 <P>aggregate function followed by timestamp field sorted in descending order did the trick:</P><P></P><P>streaming_df.groupBy("field1", "time_field").max("field2").orderBy(col("time_field").desc()).display()</P><P></P> Mon, 08 Aug 2022 20:29:41 GMT https://community.databricks.com/t5/data-engineering/display-not-updating-after-1000-rows/m-p/35042#M25727 MrsBaker 2022-08-08T20:29:41Z