Why do I see data loss with Structured streaming jobs?

brickster_2018
Databricks Employee
Databricks Employee

‎06-25-2021 05:51 AM

I have a Spark structured streaming job reading data from Kafka and loading it to the Delta table. I have some transformations and aggregations on the streaming data before writing to Delta table

Labels:
0 Kudos

brickster_2018
Databricks Employee
Databricks Employee

‎06-25-2021 05:53 AM

The typical reason for data loss on a Structured streaming application is having an incorrect value set for watermarking. The watermarking is done to ensure the application does not develop the state for a long period, However, it should be ensured the application is not losing data because of the low value of watermarking.

View solution in original post

0 Kudos