https://community.databricks.com/t5/data-engineering/why-do-i-see-data-loss-with-structured-streaming-jobs/m-p/19925#M13431 <P>The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured the application is not losing data because of the low value of watermarking. </P> Fri, 25 Jun 2021 12:53:04 GMT brickster_2018 2021-06-25T12:53:04Z https://community.databricks.com/t5/data-engineering/why-do-i-see-data-loss-with-structured-streaming-jobs/m-p/19924#M13430 <P>I have a Spark structured streaming job reading data from Kafka and loading it to the Delta table. I have some transformations and aggregations on the streaming data before writing to Delta table</P> Fri, 25 Jun 2021 12:51:31 GMT https://community.databricks.com/t5/data-engineering/why-do-i-see-data-loss-with-structured-streaming-jobs/m-p/19924#M13430 brickster_2018 2021-06-25T12:51:31Z https://community.databricks.com/t5/data-engineering/why-do-i-see-data-loss-with-structured-streaming-jobs/m-p/19925#M13431 <P>The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured the application is not losing data because of the low value of watermarking. </P> Fri, 25 Jun 2021 12:53:04 GMT https://community.databricks.com/t5/data-engineering/why-do-i-see-data-loss-with-structured-streaming-jobs/m-p/19925#M13431 brickster_2018 2021-06-25T12:53:04Z