https://community.databricks.com/t5/data-engineering/dlt-streaming-table-showing-more-quot-written-records-quot-than/m-p/87225#M37421 <P>Also after running this for a while i get these errors:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DBMIVEN_0-1725284693156.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10865iDCFC739C81BC2E09/image-size/medium?v=v2&amp;px=400" role="button" title="DBMIVEN_0-1725284693156.png" alt="DBMIVEN_0-1725284693156.png" /></span></P><P>&nbsp;</P> Mon, 02 Sep 2024 13:51:49 GMT DBMIVEN 2024-09-02T13:51:49Z https://community.databricks.com/t5/data-engineering/dlt-streaming-table-showing-more-quot-written-records-quot-than/m-p/87221#M37420 <P>Hi!</P><P>I have a DLT setup streaming data from incoming parquet files, into bronze, silver and gold tables. There is a strange bug where in the Graph gui, the number of written records for the gold streaming-table is far greater than the actual data that is written to it. I'm doing a downsampling operation on the timeseries data (for testing) when streaming to the gold table. (1 minute intervals)   </P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="1.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10860i5B8F31FF6CEBD948/image-size/medium?v=v2&amp;px=400" role="button" title="1.png" alt="1.png" /></span></P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="2.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10862iB489E4F53E862B77/image-size/medium?v=v2&amp;px=400" role="button" title="2.png" alt="2.png" /></span></P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="graph.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10863i447427E9C64557DC/image-size/medium?v=v2&amp;px=400" role="button" title="graph.png" alt="graph.png" /></span></P><P>&nbsp;</P><P>Also: I wonder if there's anything wrong with how i give Auto Loader the path to the data:</P><DIV><DIV><SPAN>data_path </SPAN><SPAN>=</SPAN> <SPAN>"/Volumes/fc_lake_catalog/test_external_schema/test_external_volume/2024/*/*"</SPAN></DIV><DIV><SPAN>the parquet files gets written&nbsp;ADLS Gen2 with EventHub in this kind of structure: 2024/08/04/23/59/*.parquet.</SPAN></DIV><DIV>&nbsp;</DIV></DIV><P>   </P> Mon, 02 Sep 2024 13:37:38 GMT https://community.databricks.com/t5/data-engineering/dlt-streaming-table-showing-more-quot-written-records-quot-than/m-p/87221#M37420 DBMIVEN 2024-09-02T13:37:38Z https://community.databricks.com/t5/data-engineering/dlt-streaming-table-showing-more-quot-written-records-quot-than/m-p/87225#M37421 <P>Also after running this for a while i get these errors:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DBMIVEN_0-1725284693156.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10865iDCFC739C81BC2E09/image-size/medium?v=v2&amp;px=400" role="button" title="DBMIVEN_0-1725284693156.png" alt="DBMIVEN_0-1725284693156.png" /></span></P><P>&nbsp;</P> Mon, 02 Sep 2024 13:51:49 GMT https://community.databricks.com/t5/data-engineering/dlt-streaming-table-showing-more-quot-written-records-quot-than/m-p/87225#M37421 DBMIVEN 2024-09-02T13:51:49Z