Why do I see data loss with Structured streaming jobs?

[User16869510359]

I have a Spark structured streaming job reading data from Kafka and loading it to the Delta table. I have some transformations and aggregations on the streaming data before writing to Delta table

[User16869510359 · Accepted solution]

The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured the application is not losing data because of the low value of watermarking.