cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Why do I see data loss with Structured streaming jobs?

User16869510359
Esteemed Contributor

I have a Spark structured streaming job reading data from Kafka and loading it to the Delta table. I have some transformations and aggregations on the streaming data before writing to Delta table

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured the application is not losing data because of the low value of watermarking.

View solution in original post

1 REPLY 1

User16869510359
Esteemed Contributor

The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured the application is not losing data because of the low value of watermarking.