cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Spark Structred Streaming, An Aggregation DF with Watermark in Append mode to Delta table is not writing the most recent aggregation to the Delta table even after crossing the water mark boundary. This is causing dataloss

KKDataEngineer
New Contributor III

Team,

 I am struggling with a unique issue. I am not sure if my understanding is wrong or this is a bug with spark.

  1.  I am reading a stream from events hub ( Extract)
  2.  Pivoting and Aggregating the above dataframe ( Transformation). This is a WATERMARKED aggregation.
  3.  writing the aggregation to Delta table in APPEND mode with a Trigger . 

However, the most recently published message to event hub is not writing to delta even after falling out of the watermark time. 

I am using the same eventtimestamp column in windowing and watermarking. 

 My understanding is the data should be inserted to the Delta table after "max of Eventtime"(latest message)+Watermark. 

This is causing a data loss.

Moreover, all the events in the memory stored must be flushed out to the sink before stopping the stream to mark a graceful shutdown.

Please advise.

0 REPLIES 0

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now