Spark Structred Streaming, An Aggregation DF with Watermark in Append mode to Delta table is not writing the most recent aggregation to the Delta table even after crossing the water mark boundary. This is causing dataloss

[KKDataEngineer]

Team,

 I am struggling with a unique issue. I am not sure if my understanding is wrong or this is a bug with spark.

1.  I am reading a stream from events hub ( Extract)
2.  Pivoting and Aggregating the above dataframe ( Transformation). This is a WATERMARKED aggregation.
3.  writing the aggregation to Delta table in APPEND mode with a Trigger . 

However, the most recently published message to event hub is not writing to delta even after falling out of the watermark time. 

I am using the same eventtimestamp column in windowing and watermarking. 

 My understanding is the data should be inserted to the Delta table after "max of Eventtime"(latest message)+Watermark. 

This is causing a data loss.

Moreover, all the events in the memory stored must be flushed out to the sink before stopping the stream to mark a graceful shutdown.

Please advise.